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ABSTRACT 


Software  Quality  Assurance  (SQA)  is  recognized  as  an 
essential  function  needed  to  monitor  the  software  system 
development  life  cycle  (SDLC).  The  inability  to 
objectively  assess  the  quality  of  the  software  system,  as 
it  is  being  developed,  has  caused  implementation  problems, 
cost  and  schedule  overruns,  and  the  delivery  of 
end-products  that  are  difficult  to  maintain  and  often  fail 
to  satisfy  user  requirements. 

The  framework  established  for  Software  Quality 
Metrics  (SQM)  provides  goal-directed  system  specifications 
and  the  ability  to  quantitatively  assess  the  quality  of  the 
system  under  development.  The  Automated  Measurement  Tool 
(AMT),  which  operationalizes  the  application  of  SQM, 
functions  as  the  core  of  a  Decision  Support  System, 
providing  quantitative  measures  and  various  levels  of 
reports. 

An  analysis  of  current  SQA  aids  enabled  the 
recommendation  of  a  minimum  set  of  tools  and  techniques  to 
be  used  by  the  SQA  program  for  monitoring  the  SDLC. 

The  SDLC  has  been  envisioned  as  an  iterative  process 
controlled  by  management.  Recognizing  the  functional 
impact  of  specific  information  as  the  key  to  objectively 

vii 


monitoring  and  controlling  the  software  system  development, 
the  decision-making  model  was  conceptualized  as  three 
subsystems  within  each  phase  of  the  SDLC:  scanning 
(afferent),  organizing  (intelligence),  and  decision 
(efferent).  Because  the  SQM  concepts  are  based  on  the 
availability  of  key  information,  the  discussion  of  the 
model  pinpointed  the  timing  for  the  appropriate  documents 
that  should  contain  specific  information  which  is  needed  to 
assess  the  software  quality.  Moreover,  the  use  of 
checklists  by  system  developers  highlights  a  prescriptive 
method  of  goal-directed  development. 

In  all,  the  thesis  provides  justification  for  using 
Software  Quality  Metrics  by  reviewing  the  need  anc. 
demonstrating  how  the  SQM  concepts  can  now  be  used  by 
AFLC/LM. 
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CHAPTER  I 


INTRODUCTION;  BACKGROUND  INFORMATION 


In  recent  years  there  has  been  an  increased  emphasis  to 
establish  Quality  Assurance  (QA)  programs  at  all  levels 
within  the  Department  of  Defense.  DoP  Directive  4155.1 
defined  Quality  Assurance  as  a  planned  and  systematic 
pattern  of  all  actions  necessary  to  provide  adequate 
confidence  that  material,  data,  supplies,  and  services 
conform  to  established  technical  requirements  and  achieve 
satisfactory  performance  (Ref  37). 

The  use  of  MIL-S-52779A,  Software  Quality  Assurance 
Program  Requirements,  in  DoD  contracts  (requiring 
government  contractors  to  establish  a  software  QA 
function),  has  generated  an  increased  awareness  of  the  need 
for  QA  in  the  development  of  software;  yet  many  DoD 
organizations  have  been  unable  to  internally  establish 
Software  Quality  Assurance  (SQA)  programs  (Ref  52).  The 
Air  Force  Logistics  Command  ( AFLC )  has  recognized  the  need 
for  a  SQA  program  mainly  because  of  the  cost  of  producing 
and  maintaining  its  wide  spectrum  of  conventional  software 
systems;  Management  Information  Systems  (MIS)  for 
Plans/Programs/Budget,  Data  Base  systems.  Inventory 
systems,  Accounting  systems,  and  Automated  Logistics 


Maintenance  systems  (Ref  38).  AFLC  has  recognized  that 
"maintenance  costs  for  software  range  from  50-75%  of  total 
system  costs,  thus,  the  command  position  is  to  design  and 
develop  software  that  i3  easy  to  maintain  in  order  to 
reduce  operational  phase  costs"  (Ref  89). 

Organizationally,  AFLC  has  six  main  users  of 
conventional  software  systems:  Procurement  &  Management 
(PM),  Personnel  (MP),  Logistics  Operations  (LO), 
Maintenance  (MA),  Comptroller  (AC),  and  Logistics 
Management  (LM).  AFLC/LM  is  responsible  for  software 
development  within  AFLC.  It  provides  the  Project 
Management  ( LMX ) ,  Configuration  Management  and  Project 
Engineering  (LME),  Technical  Support  ( LMT ) ,  ADP  Resources 
(LMD),  Computer  Operations  (LMO),  and  Training  ( LMR )  for 
the  development  of  software  systems.  Once  the  system  is 
operational,  it  is  "turned-over"  to  the  user  of  the  system, 
i.e.,  AFLC/LO  (Ref  48). 

Software  Life  Cycle  Iirplications 

AFLC  recognizes  that  by  implementing  a  SQA  program  it 
will  be  enhancing  the  software  development  by  imposing  a  QA 
discipline  on  the  software  life  cycle.  The  Software 
Development  Life  Cycle  (SDLC),  as  described  in  Dod  Standard 
7935. 1-S,  provides  for  the  structured  implementation  of 
software  from  conceptual iza tion  through  operations 
(Ref  47). 

Represented  in  Figure  1,  the  SDLC  begins  with  the 
conceptualization  or  Requirements  Analysis.  This  phase 
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determines  "what"  the  software  system  is  to  do.  Next  the 
Design  phase  describes  "how"  the  requirements  will  be  met. 
Following  this  is  the  Programming  (Coding)  and  Checkout 
phase,  which  entails  the  greatest  amount  of  manpower  and 
funds.  While  the  test  phase  concludes  the  normal 
development  cycle,  one  should  realize  that  with  software 
the  development  continues  in  the  Operations  and  Maintenance 
phase  because  of  debugging  and  changes  in  the  user 
requirements  (Ref  11). 


Figure  1. 

Software  Development  Life  Cycle  (SDLC) 


The  arrows  in  Figure  1  indicate  the  possible  paths  of 
software  development.  They  demonstrate  that  the 
development  phases  are  iterative  in  nature.  The  iterations 
show  that  the  identification  of  problems  causes  the 
development  effort  to  "fail-back"  on  to  earlier  phases  in 
order  to  correct  the  problem.  One  should  bo  aware  that 
there  are  associated  costs  for  such  problems  or  errors. 
Figure  2  reveals  that  the  earlier  an  error  is  made,  t he 
later  it  will  be  detected  in  the  SDLC  (Kef  14).  Thus,  it 
costs  more  to  correct  errors  made  during  i  h*'  requirements 
analysis  than  it  does  For  errors  made  in  cod i m  because 


e  r  r  o  r 


use  of 


later  detection 


of  an 


pqu-it 


i  n  t  he 


ERRORS  INTRODUCED 


ERROR  DETECTED 


Figure  2, 

A  Software  Development  Problem 
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additional  time  and  resources.  Furthermore, 


extensive 


analysis  of  error  data  by  TRW  reveals  that  most  errors  are 
design  and  requirements  errors,  as  opposed  to  coding 
errors  (Ref  78). 

With  these  facts  in  mind,  it  would  appear  logical  to 
assume  that  any  tool  that  can  improve  the  quality  of  the 
requirements  analysis  would  have  a  significant  impact  on 
the  overall  development  cost.  Moreover,  it  follows  that 
there  would  be  a  genuine  attempt  to  find  ways  of  improving 
quality,  hence  reducing  the  overall  costs  by  saving  time 
and  resources.  However,  as  of  this  date,  AFLC  uses  no 
quality  measurement  tools  during  the  requirements  analysis 
or  design  phase.  The  consequence  of  this  vacuum  created  by 
the  lack  of  up-front  measurement  has  been  the  delivery  of 
software  systems  that  must  be  modified  in  order  to  meet 
user  needs  (Ref  38). 

AFLC1 s  Software  Development  Efforts 

Efforts  to  develop  quality  assurance  programs  for 
software  systems  at  AFLC  began  in  1974  with  the  Advance 
Logistics  System  (ALS).  This  was  designed  to  integrate  the 
logistic  software  systems.  However,  because  ALS  was  late 
in  being  developed  at  a  cost  that  had  far  exceeded  its 
budgeted  amount,  AFLC  was  forced  to  firm  up  the  software 
management  discipline. 

The  likely  candidate  to  help  accomplish  this  task  was 
Configuration  Management  (CM)  because  it  had  been  used  for 
years  in  the  development  of  weapons  systems  and,  as  a 
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result,  people  thought  that  it  could  be  transferred  to 
software  development.  A  "get-well"  plan  was  implemented 
which  brought  in  CM  people  from  the  weapons  SPOs  (System 
Program  Office)  to  create  an  ALS  SPO  (Ref  42). 

In  July  1975  the  new  ALS  SPO  developed  AFLCR  300-3,  a 
document  which  at  that  time  was  viewed  as  being 
"on-the-cutting-edge "  for  software  configuration 
management.  Unfortunately,  the  new  regulation  was 
developed  too  late  to  improve  the  development  management  of 
ALS;  so  in  December  1975  Congress  killed  ALS.  As  a  result, 
the  development  of  software  configuration  management  was 
halted  for  several  years  after  that  (Ref  42). 

In  January  1978  AFR  300-15  (built  on  AFLCR  300-3  and 
AFDSDC  300-8)  was  published  (Ref  47).  Although  this 
document  includes  a  one  page  chapter  entitled  "Quality 
Assurance",  it  falls  short  of  insuring  a  quality  software 
product.  While  the  regulation  says  that  quality  assurance 
will  exist,  it  fails  to  specify  what  such  a  program  should 
involve.  This  is  evidenced  by  the  fact  that  AFR  300-15, 
like  the  other  regulations  that  govern  software 
development,  makes  no  attempt  to  define  or  measure  quality. 

Within  AFLC/LM,  quality  assurance  is  supposed  to  be  a 
function  within  Configuration  Management,  but.  in  reality, 
"CM  fails  to  provide  software  QA. "  This  sentiment  of 
"having  QA"  contributes  to  a  lack  of  urgency.  Moreover, 
because  the  CM  people  are  more  hardware  oriented,  they  lack 
the  skills  for  software  QA  (Ref  42).  In  other  words. 
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weapons  systems  (hardware)  CM  has  not  readily  transferred 
to  software  developmf.it. 


Failures  of  Current  Development  Efforts 

Another  factor  which  inhibits  an  effective  Quality 
Assurance  effort  under  the  current  AFLC/LM  structure  is  the 
dependence  between  Configuration  Management  and  Program 
Management  (Ref  42).  Ray  Rubey  from  SOFTEC  calls  this 
"Cognitive  Dissonance"  because  individuals  are  asking 
people  to  report  where  they  went  wrong  when  they  truly 
believe  that  they  did  a  good  job  (Ref  44).  In  other  words, 
SQA  should  be  an  independent  effort  by  a  functionally 
separate  team  (Ref  52).  Moreover,  a  formal  SQA  program 
would  indicate  commitment  by  management  to  the  requirements 
for  a  quality  operation,  and  it  would  make  SQA  a  visible 
and  identifiable  function  (Ref  38). 

In  November  1981,  at  a  symposium  at  AFIT,  Ray  Rubey 
identified  some  of  the  reasons  why  SQA  can  fail  (Ret  44). 
Four  of  the  five  reasons  would  seem  to  directly  apply  to 
AFLC/LM  in  software  systems  development  effort: 

1.  QA  is  not  included  in  the  budget.  Although  a 
"flavor"  of  QA  is  expected  from  CM,  no  dollars  are  directly 
allocated  for  SQA  (Refs  38,  42).  Quality  Assurance  costs 
include  QA  personnel  salaries,  administrative  costs, 
computer  time,  etc.  If  resources  or  programs  are  not 
budgeted  at  the  beginning  of  a  project,  it  is  difficult  to 
expect  that  funds  will  later  be  found  for  them. 

2.  Inexperienced  or  incompetent  people  are  on  QA 


teams.  AFLC/LM  personnel  admitted  that  CM  people  are  not 
trained  in  SQA  (Ref  42). 

3.  QA  does  not  start  when  the  development  starts. 
Currently,  even  CM  reviews  do  not  occur  until  after  the 
requirements  are  specified,  and  there  is  no  SQA  effort  ir 
AFLC/LM  (Refs  38,  89). 

4.  QA  and  system  objectives  are  unknown.  CM  does  not 
even  address  the  basic  question  —  What  do  we  mean  bj 
quality?  The  objectives  and  quality  factors  are  not 
identified  (Ref  38).  AFR  300-15  does  state  that  "QA 
policies  and  procedures  must  be  set  up  which  tell  how  to 
conduct  configuration  audits,  and  how  to  make  sure  that  the 
CPCI  meets  ADP  standards..."  Yet  if  the  policies, 
procedures,  and  standards  fail  to  mention  anything  aboub 
measuring  quality  (which  is  the  case),  then  simply 
complying  with  them  also  fails  to  insure  quality  software. 
Richard  Woodward  in  AFLC/LM,  the  sponsor  of  this  research, 
has  indicated  that  CM  fails  to  provide  standards  for 
acceptance  and  that  under  CM  there  is  no  wa>  to  determine 
the  quality  of  the  software  system.  clearly,  "aflc 
currently  has  no  command  level  Quality  Assurance  program 
for  software"  (Ref  38). 

Currently,  there  is  a  lack  of  clear  evidence  that  SQA 
is  a  reality,  despite  the  attempt  of  AFR  300-15  to  claim  QA 
is  in  place.  Indeed,  defenders  can  point  to  phrases  that 
imply  standards  must  be  followed,  but  which  standards  are 
to  be  followed?  In  addition  to  shortcomings  already 
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pointed  out,  AFR  300-15  provides  no  criteria  for 
determining  which  standards,  if  any,  should  be  used. 
Personnel  in  AFLC/LM  noted  that  there  is  no  directive  that 
sets  one  standard  for  software  development  for  AFLC/LM 
there  are  several  different  standards,  and  none  are 

quantified  (Ref  48). 

Although  Configuration  Management  has  failed  to  provide 
the  standards  for  acceptance,  and  hence  failed  to  quantify 
quality,  it  does  provide  the  framework  on  which  a  SQA 
program  can  be  built  because  it  establishes  the  structure 
necessary  to  enforce  compliance  with  procedures.  Using 
DODD  7935. 1-S,  AFR  300-12,  and  AFR  300-15,  AFLC/LME 
described  the  SDLC,  as  depicted  in  Figure  3,  to  provide  a 
structured  model  for  software  development 

(Ref 8  46,  47,  85).  Because  this  structure  governs  the 
software  development  environment  for  AFLC/LM,  it  is 
presented  in  more  detail  in  Appendix  F  which  provides  a 
listing  of  all  deliverable  documents  for  each  milestone 
(review)  in  the  SDLC  (Ref  89).  A  glossary  of  terms  for  the 
software  Configuration  Management  follows  in  Appendix  G 
which  provides  a  description  of  each  review  and  a 
definition  for  CM  terms. 

Current  SQA  Efforts 

With  a  development  .structure  firmly  in  place,  AFLC/LM 
has  begun  to  define  requirements  for  a  Software  Quality 
Assurance  Program.  a  formal  SQA  program  would  indicate 
commitment  by  management  to  the  requirements  for  a  quality 
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Figure  3.  Configuration  Management  for  AFLC 


operation.  It  would  also  make  SQA  a  visible/identifiable 
function  to  management  and  to  the  development 
staff  (Ref  38).  Initially,  three  alternatives  to  the  SQA 
program  were  proposed  by  AFLC/LM. 

The  first  alternative,  called  the  "Broad  Program," 
would  have  used  a  quality  audit  approach  applied  to  all 
systems  at  varying  levels.  At  the  higher  level  would  be  a 
consideration  of  compliance  with  DoD/Air  Force  directives 
and  a  checking  for  effective  operational  interface.  The 
lower  level  would  get  more  into  specifics,  evaluating 
performance  against  designated  standards  and  requirements. 

The  second  alternative,  labeled  "Main  Projects,"  would 
have  implemented  a  quality  assurance  program  that  would 
subject  major  projects  to  tailored  evaluations  or 
considerations.  Evaluations  then  could  be  made  that  were 
specific  to  each  particular  system.  Systems  not  classified 
as  major  would  be  subjected  to  a  less  vigorous  review  or 
audit.  Although  this  alternative  would  be  the  least  costly 
of  the  three  alternatives,  it  would  probably  also  be  the 
least  effective. 

The  third  alternative,  called  the  "Cradle  to  Grave" 
quality  program  implies  QA  evaluations,  such  as  checklists 
and  standards,  throughout  the  life  cycle  of  all  systems. 
It  is  designed  to  involve  users  of  the  system,  require 
documenting  of  all  significant  actions,  promise  error 
tracking  and  auditing  capabilities,  and  include  quality 
checks  for  systems  interfacing  with  the  basic  system.  The 
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plan  states  that  documentation  and  related  actions  for  each 
phase  must  be  evaluated  and  accepted  by  a  QA  task  group 
before  proceeding  to  the  next  phase  and  before  releasing 
the  system  for  implementation. 

Because  the  third  alternative  includes  projects  of  all 
sizes  and  would  operate  throughout  the  total  life  cycle  of 
software  development,  it  will  be  adopted  as  AFLC's  Software 
Quality  Assurance  Program  (Ref  38). 

One  of  the  basic  requirements  for  the  success  of  this 
SQA  program  is  that  a  "carefully  prepared  acceptance 
checklist  should  be  developed."  (Ref  38)  To  implement  such 
a  checklist,  tools  are  needed  to  quantify  the  quality  of 
the  software  under  development.  Fortunately,  these  tools 
are  now  available  through  certain  sectors  of  industry. 

Measuring  Software  Quality 

Recognizing  the  importance  of  such  tools,  DoD  has 
recently  started  funding  efforts  which  are  attempting  to 
adapt  industrial  research  to  military  requirements.  This 
research  has  resulted  in  the  development  and  evaluation  of 
a  number  of  metrics  purporting  to  measure  various 
qualitative  attributes  of  software.  These  metrics,  once 
established,  can  provide  quantifiable  measures  for  software 
quality  (Ref  23). 

The  metric  measurements  are  derived  from  implied 
standards  which  are  accepted  as  being  necessary  for  quality 
software.  Basically,  the  metrics  are  a  check  on  the 
development  procedures  to  determine  if  certain  standards 
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were  incorporated  (regardless  if  they  were  stated  or  not). 
Justification  for  SQA 

The  need  for  software  quality  assurance  has  been 
demonstrated  in  a  Governmental  Accounting  Office  (GAO) 
report  which  reviewed  nine  federal  software  projects 
totaling  $6.75  million.  This  total  included: 

$3.20  M  of  software  that  was  delivered,  but  never  used, 

$1.95  M  of  software  that  was  paid  for,  but  not  delivered, 

$1.30  M  of  software  that  was  used  temporarily,  then 

abandoned, 

$  . 20  M  of  software  that  was  used  only  after  changes 
were  made, 

$  . 10  M  of  software  that  was  used  as  it  was 
delivered  (Ref  40). 

In  other  words,  less  than  1.5%  of  the  total  funds  spent  on 
software  actually  produced  software  Products  that  could  be 
used  by  the  user  as  delivered.  The  report  suggested  that 
if  measurable  standards  were  applied  during  the  systems 
development  life  cycle,  then  the  user  of  the  software  could 
be  better  assured  of  receiving  a  usable  software  system. 

In  recent  years,  symptoms  indicating  the  existence  of 
an  inadequate  quality  assurance  program  have  been 
encountered  in  the  development  of  large  scale  systems. 
These  symptoms  include:  cost  and  schedule  overruns,  poor 
performance  of  the  systems  once  they  are  delivered,  high 
maintenance  costs,  lack  of  reliability,  and  a  high  degree 
of  system  sensitivity  to  changes  in  requirements.  Examples 
of  these  situations  have  been  well  documented  in  the 
literature  (Refs  1, 2 , 3, 4, 5, 6 , 7  ) .  In  short,  government  and 
industry  sources  have  identified  the  enforcement  of 
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quantifiable  standards  throughout  the  system  development 
life  cycle  as  one  key  to  delivering  quality  software 
systems. 

Conceptualization  of  the  Decision-Making  Process 

In  order  to  understand  why  current  software  systems 
development  efforts  have  failed  to  enforce  measurable 
standards,  one  has  to  have  an  adequate  conceptualization  of 
the  decision-making  process  which  is  involved  in  creating  a 
complete  software  system.  In  Figure  4,  the  software 
development  process  is  represented  as  an  open  system  with 
relationships  between  its  own  internal  organization  and  the 
environment  (Ref  49). 

During  the  software  system  development  effort  for 
AFLC/LM,  Configuration  Management  provides  the  structure 
for  the  three  subsystems:  Scanning,  Organizing,  and 
Decision.  Conceptually,  the  actions  of  AFLC's  system 
development  organization,  which  are  the  outputs  of  the 
decision  subsystem,  should  not  be  based  on  the  outputs  of 
the  scanning  subsystem,  but  rather  upon  the  outputs  of  an 
intelligence  (organizing)  subsystem  which  would  act  as  an 
evaluator  of  the  receptor  or  scanning  subsystem.  Data 
received  by  the  scanning  (receptor)  system  needs  to  be 
eventually  fed  into  the  decision  system  where  it  can  be 
utilized  for  problem-solving  purposes  (Ref  49). 

It  is  the  data  which  is  currently  collected  that  makes 
the  eventual  decisions  ineffective.  The  current  scanning 
(receptor)  data  is  the  result  of  checking  for  compliance 
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with  outputs,  i.e.,  to  check  if  source  documents  are  being 
produced  to  monitor  the  software  development;  yet  the 
needed  information  should  check  the  quality  of  that 
documentation.  AFLC  personnel  noted  that  although  CM 
insures  that  the  documents  are  produced,  that  there  is  no 
check  to  insure  the  quality  of  the  document  (Ref  48). 

A  software  QA  program  can  provide  the  tools,  such  as 
Software  Metrics,  which  would  indicate  the  quality  of  the 
documentation  being  produced.  This  is  precisely  why  the 
data  collected  by  such  a  SQA  program  would  be  inherently 
different  from  that  collected  under  CM.  Such  metric  data 
would  indicate  the  quality  of  the  documentation  rather  than 
just  its  existence.  By  using  Software  Metrics  the 
organizing  (intelligence)  subsystem  would  be  able  to  relate 
the  data  into  meaningful  measures  which  are  then  passed  on 
to  the  decision  subsystem.  This  model  of  the 
decision-making  process  of  the  software  development  effort 
will  be  used  in  Chapter  5  to  demonstrate  how  software 
metrics  can  be  applied  by  AFLC/LM. 

Front-End  Planning  Through  Configuration  Management 

Additionally,  Configuration  Management  provides  a 
planning  structure  to  the  development  effort.  Tn  the 
development  of  a  software  system,  AFLC  needs  to  specify  a 
system's  requirements,  and  then  be  able  to  determine 
whether  those  system  requirements  are  being  satisfied  as 
the  software  system  evolves.  The  parameters  of  the 
specification  center  around  the  technical  definition  of  the 
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application  and  the 

software 

role 

within  the 

overall 

system.  CM  aids  in 

this 

front 

-end 

planning  and  could 

greatly  increase  the 

software 

management  effort 

if  it 

incorporated  a  definition 

of 

quality  and  the 

system 

objectives  (Ref  42). 

While  the  application 

functions , 

cost,  and 

schedule 

aspects  of  development  can  be  objectively  defined, 
measured,  and  assessed  throughout  the  development  of  the 
system,  the  quality  desired  has  historically  been  definable 
only  in  subjective  terms.  This  occurs  because  there  are  no 
quantif iable  criteria  against  which  to  judge  the  quality  of 
the  software  until  the  system  is  under  operational 
conditions  (Ref  38).  Current  scheduling  methods,  such  as 
PERT/CPM  or  GANTT  charts  tell  nothing  about  the  quality  of 
the  system.  Rather,  they  provide  information  about  the 
state  of  existence,  i.e.,  is  the  software  completed  by  a 
certain  time  within  a  specified  cost  (Refs  43,  45). 

A  Framework  for  Software  Quality  Assessment 

Fortunately,  there  is  a  framework  for  software  quality 
metrics  that  can  provide  the  measurement  tools  needed  to 
quantitatively  assess  software  quality.  This  framework  has 
evolved  over  a  number  of  years.  Boehm  traces  some  of  the 
previous  work  contributing  to  this  evolution  (Ref  19).  The 
work  dates  back  to  a  paper  by  Ray  Rubey  and  R.  Hart  wick  jn 
1968  which  first  introduced  the  concept  of  software  metrics 
(Ref  20).  Later  studies  established  a  more  formal 
conceptual  framework  (Refs  21,  22). 
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The  catalyst  that  set  off  this  attempt  to  quantify 
attributes  of  software  was  the  introduction  of  more  formal 
and  structured  software  design,  implementation,  and  review 
techniques  —  a  framework  which  was  conducive  to  the 
quantitative  measurement  of  software  quality  (Ref  31). 
Figure  5  illustrates  the  hierarchical  nature  of  this 
framework. 

At  the  highest  level,  the  major  aspects  (factors)  of 
software  quality  are  identified.  The  user  is  the  program 
manager  or  the  customer  of  the  software  system.  The  user 
requires  a  defined  set  of  factors  in  order  to  identify  what 
qualities  are  desired  in  the  software  system  being 
developed.  To  satisfy  this  use,  the  definitions  of  the 
factors  must  lend  themselves  to  quantification 
(measurement)  that  is  meaningful  to  users. 

The  approach  taken  to  satisfy  these  requirements  is  to 
evaluate  how  a  program  manager  views  the  end  product  of  a 
software  development.  The  orientations  or  viewpoints 
identified  relate  to  life  cycle  activities  involving  the 
software  product  (Ref  18). 

Underlying  these  user-oriented  quality  factors  is  a  set 
of  attributes  (criterion)  which,  if  present  in  the 
software,  provide  the  characteristics  represented  by  the 
factors.  For  each  factor  then,  a  set  of  criteria  has  been 
established  and  defined  (Ref  16). 

A  key  point  in  the  approach  is  that  the  measurements 
are  to  be  taken  during  the  development  effort.  These 
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measurements  are  not  post-implementation  assessments  of 
software  quality.  Their  purpose  is  to  provide  an 
indication  of  the  progression  toward  a  desired  level  of 
quality.  The  set  of  attributes,  or  criteria,  established 
for  each  quality  factor  then  represents  attributes  which 
can  be  measured  during  the  software  development  (Ref  17). 

The  framework  provides  a  mechanism  for  a  project 
manager  to  identify  what  qualities  are  important.  These 
qualities  are  attributes  of  the  software  in  addition  to  its 
functional  correctness  and  performance  which  have  life 
cycle  implications  (Ref  28).  Such  factors  as  reliability 
and  maintainability  have  been  shown  in  recent  years  to  have 
significant  life  cycle  cost  impacts  (Ref  24). 

Background  Abstract 

Because  of  the  significant  cost  and  schedule  overruns, 
DoD  has  tried  to  place  an  emphasis  on  QA  programs  through 
Military  Standards,  yet  AFLC  has  been  unable  to  incorporate 
these  in  the  development  of  its  conventional  software 
systems.  The  quantitative  nature  of  software  metrics 
provides  the  measurement  tool  needed  in  a  SQA  program  to 
determine  the  qualitative  attributes  of  software.  By 
having  a  tool  by  which  to  objectively  measure  software 
quality,  AFLC/LM  will  be  able  to  enhance  the 
controllability  of  software  development  effort  by 
quantifying  the  information  used  in  SDLC  decision-making. 

Tn  summary,  AFLC  currently  has  no  quantifiable  measures 
for  software  quality.  Even  though  they  have  tried  and  are 
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continuing  to  try  to  define  programs  to  develop  quality 
software,  AFLC/LM  personnel  have  not  been  able  to  implement 
the  measurement  tools  that  can  provide  quantifiable 
quality  (Ref  38). 


Problem  Statement 

AFLC  needs  a  software  quality  assurance  (SQA)  program 
that  can  quantitatively  measure  the  quality  of  computer 
software  during  all  stages  of  its  development.  Current 
attempts  to  establish  such  a  program  have  been  frustrated 
by  program  management' s  inability  to  define  the  critical 
information  needed  to  operationalize  such  a  program. 


Research  Objectives 


The  overall  objective  of  this  study  is  to  provide 
recommended  guidelines  to  AFLC/LM  concerning  ways  to 
objectively  specify  and  quantitatively  measure  the  desired 
amount  of  quality  in  a  computer  software  system  during  all 
stages  of  its  development.  To  accomplish  this,  the 
following  four  goals  had  to  be  attained: 

1.  Determine  which  measurement  tools  and  techniques 
should  be  used  by  AFLC/LM  when  it  establishes  its  Software 
Quality  Assurance  (SQA)  program. 

2.  Develop  a  model  of  the  decision-making  process  of 
the  software  system  development  effort  that  incorporates 
the  application  of  Software  Quality  Metrics  and  which 
illustrates  the  feasibility  of  implementing  a 
quantitatively  oriented  SQA  program  at  AFLC/LM. 
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3.  Demonstrate  how  the  SQM  concepts  can  be  applied  to 
existing  systems  at  AFLC  by  developing  checklists  for 
system  developers. 

4.  Use  the  information  generated  from  this  research  to 
help  AFLC/LM  formulate  guidelines  concerning  ways  that 
Software  Quality  Metrics  can  be  integrated  into  a 
command-wide  SQA  program. 

Scope  of  Research 

This  research  effort  concentrated  on  recommending 
guidelines  for  a  system  to  control  and  monitor  the  software 
development  process  within  AFLC. 

An  attempt  has  been  made  to  identify  the  quality 
metrics  and  information  requirements  of  an  effective 
software  management  program.  Since  reliability  and 
maintainability  were  identified  as  the  two  most  critical 
factors  in  establishing  a  SQA  program  for  AFLC  (because 
AFLC  must  rely  on  the  software  systems  for  several  years), 
this  thesis  indicates  how  a  software  system  can  be  measured 
to  insure  reliability  and  maintainability.  It  is 
anticipated  that  by  enhancing  AFLC's  ability  to  measure  the 
reliability  and  maintainability  attributes  of  software 
quality  that  a  marked  improvement  in  the  control  of 
software  development  will  occur. 

AFLC/LM,  the  sponsor  of  this  research  effort,  has  been 
the  central  point  for  gathering  data  about  AFLC's  current 
and  past  attempts  at  quality  assurance.  Data  used  in 
discussing  the  metrics  was  obtained  from  the  Rome  Air 
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Development  Center  (RADC)  at  Griffis  AFB,  NY.  The  data  was 
taken  from  USAP  systems  that  are  similar  to  those  at  AFLC. 

Additionally,  the  metric  worksheets  have  been  applied 
to  G072A,  a  subsystem  of  the  Maintenance  Management  Systems 
Improvement  Project  (MMSIP)  to  determine  the  changes  needed 
by  APLC  to  be  able  to  use  the  software  metrics. 

Because  most  of  AFLC's  software  development  is  a 
product  of  "in-house a  projects,  and  because  AFLC/LM  is  the 
OPR  for  conventional  computer  systems  (not  embedded 
systems),  this  research  will  provide  recommendations  to 
AFLC/LM. 

Chapter  II  provides  an  overview  of  Software  Quality 
Metrics  which  can  provide  the  quantification  needed  to 
measure  standards  in  a  SQA  program. 
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CHAPTER  II 


SOFTWARE  QUALITY  METRICS:  AN  OVERVIEW 


Quality  Measurement  in  Perspective 

The  evolution  during  the  past  decade  of  modern 
programming  practices,  structured  development  techniques 
and  methodologies,  and  requirements  for  effective 
documentation,  has  increased  the  feasibility  of  effective 
measurement  of  software  quality  (Ref  16). 

However,  before  the  potential  of  measurement  techniques 
could  be  realized,  a  framework  or  model  of  software  quality 
had  to  be  constructed.  An  established  model,  which  at  one 
level  provides  a  user  or  management-oriented  view  of 
quality,  is  used  to  establish  software  quality  requirements 
for  a  specific  application  (Ref  17). 

The  actual  measurement  of  software  quality  is 
accomplished  by  applying  software  metrics  to  the 
documentation  and  source  code  produced  during  the  SDLC. 
These  measurements  are  part  of  the  established  model  of 
software  quality,  and  through  that  model  they  can  be 
related  to  various  user-oriented  aspects  of  software 
quality. 
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The  metrics  can  be  classified  according  to  three 
categories:  anomaly-detecting,  predictive,  and  acceptance. 


Anomaly-detecting  metrics  identify  deficiencies  in 
documentation  or  source  code.  These  deficiencies  usually 
are  corrected  to  improve  the  quality  of  the  software 
product.  Standards  enforcement  is  a  form  of 
anomaly -detecting  metrics. 

Predictive  metrics  are  measurements  of  the  logic  of  the 
design  and  implementation.  These  measurements  are 
concerned  with  form,  structure,  density,  and  complexity 
type  attributes.  They  provide  an  indication  of  the  quality 
that  will  be  achieved  in  the  end  product,  based  on  the 
nature  of  the  application,  and  design  and  implementation 
strategies. 

Acceptance  metrics  are  measurements  that  are  applied  to 
the  end  product  to  assess  the  final  compliance  with 
requirements.  Tests  are  a  form  of  acceptance-type 
measurements  (Ref  23). 

Complementary  Use  of  Metrics  with  SQA 

The  measurement  concepts  of  software  metrics  complement 
current  Quality  Assurance  and  testing  practices.  They  are 
not  a  replacement  for  any  current  techniques  utilized  in 
normal  QA  programs.  Table  I  summarizes  how  software 
metrics  complement  QA  activities  by  mapping  the  impacts  of 
software  quality  metrics  onto  functions  of  a  SQA 
program  (Ref  16).  For  example,  a  major  objective  of  QA  is 


Table  I 

How  Software  Metrics  Complement  Quality  Assurance 


QUALITY  ASSURANCE 

PROGRAM  REQUIREMENTS 

IMPACT  OF  SOFTWARE  QUALITY 

METRIC  CONCEPTS 

Assures  conformance  with 
requirements 

Expands  software 
requirements 

Identify  software 
deficiencies 

Expands  deficiency  detection 
with  some  metrics  designed 
specially  for  anomaly-detection 

Provide  configuration 
management 

_  -- 

Provides  measures  to  determine 
the  adequacy  of  standards  in  use 
by  quantifying  the  quality  of 
products  which  comply  with  current 
procedures 

Conduct  tests 

Assists  in  evaluation  of 
other  qualities 

Provide  library  controls 

Can  provide  measurements  to 
determine  the  adequacy  of 
standards  in  use 

Review  computer  program 
design 

Provides  metrics  that  can  predict 
end-product  capabilities 

Assure  software  documenta¬ 
tion  requirement 
conformance 

.  _ j 

Provides  metrics  that  assist  in 
evaluation  of  documentation  as 
as  well  as  code 

Conduct  reviews  and  audits 

Provides  procedures  for  applying 
metrics  in  the  form  of 
worksheets;  thus,  formalizing 
the  inspection  process 

Provide  tools/techniques/ 
methodology  for  quality 
assurance 

Expanded  state-of-the-ar t  tools, 
techniques,  and  methodol lgies 

Provide  subcontractor 
contro l 

Can  provide  measurements  on 
which  to  base  acceptance  of 
de liver ies 
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to  assure  conformance  with  user /customer  requirements.  The 
software  quality  metric  concepts  provide  a  methodology  for 
the  user/customer  to  specify  life-cycle-oriented  quality 
requirements ,  usually  not  considered,  and  a  mechanism  for 
measuring  if  those  requirements  have  been  attained. 

A  function  usually  performed  by  quality  assurance 
personnel  is  a  review/audit  of  software  products  produced 
during  a  software  development.  The  software  metrics  add 
formality  and  quantification  to  these  document  and  code 
reviews  (Ref  28). 

More  importantly ,  the  framework  provides  a  means  of 
quantitatively  assessing  how  well  the  development  is 
processing  relative  to  the  quality  goals  established.  This 
augments  current  techniques  used  by  quality  assurance 
personnel  which  may  range  from  only  testing  to  testing  and 
standards  enforcement,  participation  in  walkthroughs,  etc. 

Testing  is  usually  oriented  toward  correctness, 
reliability,  and  performance.  The  metrics  assist  in  the 
evaluation  of  other  quality  factors  like  maintainability, 
portability,  and  flexibility  (Ref  23).  The  periodic 
application  of  the  metrics  during  a  large-scale  software 
development  effort  can  be  viewed  as  a  control  system. 
Snapshot  assessments  are  generated  and  feedback  to  the 
program  management  is  provided  with  respect  to  their 
specified  requirements  for  quality,  thereby  allowing 
corrective  action,  calibration,  redirection,  or 
identification  of  areas  to  be  emphasized  later  in  the 


development,  i.e.,  such  as  during  testing. 

The  metrics  may  be  in  the  form  of  a  checklist  used  to 
"grade"  a  document  produced  during  the  development  of 
specific  attributes  such  as  the  number  of  paths  through  a 
module  or  the  number  of  unconditional  branches  (Ref  23). 
In  short.  Software  Quality  Metrics  provide  the 
quantification  needed  in  a  SQA  program. 

Comparison  of  Metrics,  Inspections,  and  Walk-Throughs 

Software  metrics  offer  more  quantification  of  software 
quality  than  other  SQA  tools  and  techniques.  Software 
metrics,  code  inspections,  and  structured  walk-throughs 
have  all  been  developed  to  aid  the  SQA  function.  Metrics 
are  used  throughout  the  SDLC  and  quantitatively  measure  the 
quality  of  the  software  as  it  is  developed  (Ref  52).  Code 
inspection  techniques  are  primarily  for  finding  errors  in 
design  and  code  (Ref  29).  The  primary  purpose  of 
structured  walk-throughs  is  to  subject  the  design  or  code 
to  a  critical  evaluation  (Ref  30). 

Currently  AFLC  uses  code  inspections  and  structured 
walk-throughs.  Although  the  level  of  usage  for  these 
techniques  varies  for  each  project,  these  techniques  fail 
to  quantify  quality  even  when  both  are  used  to  their 
fullest  potential.  Table  II  exposes  the  gaps  in  current  QA 
techniques  by  comparing  the  properties  of  Code  Inspection, 
Structured  Walk-Throughs,  to  Software  Quality 
Metrics  (Ref  23). 

Although  Code  Inspection  compares  favorably  to 
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TABLE  SI 

Comparison  of  Ray  Properties  of  Inspections  and  Walk-Througha  and  Metrics 

rtopsttUi 

Inspection 

Walk-Thru 

Metr ics 

1.  Foraal  Moderator  Training 

Yea 

No 

No 

2.  Definite  Participant  Rolea 

Yea 

No 

Yea 

3.  Who  "Drives"  The  Inap.  or 
Walk-Thm 

Moderator 

Owner  of 
Material 
(Designer  or 
Coder 

Quali'.y 

Asaur  ince 

Croup 

4.  Dae  ’Mow  to  Find  Krrora" 
Cbackliata 

Ttf 

No 

Yea 

3.  3ae  Dia  trihut  ion  of  Error 
Types  to  Look  For 

Y«l 

No 

Yes 

6.  Follow-Op  to  Reduce  Bad  Fixea 

Yea 

No 

Yea 

7.  Leaa  Future  Errora  Because  of 
Detailed  Error  Feedback  to 
Individual  Programmer 

Yes 

Incidental 

Yes 

8.  Improve  Inspection  Efficiency 
Fron  Analysia  of  Rcaulta 

Yes 

No 

Yea 

9.  Analysis  of  Data  Process 
Problens  I^>roveaent 

Yea 

No 

Yes 

10.  Lifncycle  Inpact  and 
Applicability? 

Partial 

No 

Yes 

11.  Quantification  of  Results 

For  Comparative  Purposes 

No 

No 

Yes 

12.  Prediction  of  Quality  Level 
Based  on  Current  Analysis 
and  Figure  of  Merit? 

No 

No 

Methodology 

Exists 

13.  Forail  Definition  of  Quality 
(Factors,  Attributes)? 

No 

No 

Yes 

14.  Foraal  Validation  of  Concept 
Carried  Out? 

Partial  (lack 
of  quantifiable 
results  makes  it 
dif  f  icu  It  to 
statistically 
validate 

No 

Yes 

Yes 

Yes 

15.  Foraal  Methodology  for 
Application  Developed? 

Yea 

No 

16.  Applicable  in  Different 
Environnents 

Yea 

Yes 

_ 

Structured  Walk-Throughs,  it  fails  to  provide 
quantification  of  software  quality.  The  quantification  of 
quality  is  the  most  favorable  attribute  of  Software  Quality 
Metrics  in  addition  to  it  matching  the  other  properties  of 
Code  Inspection  and  Structured  Walk-Throughs. 

Software  metrics  have  both  anomaly-detecting  and 
predictive  characteristics,  in  addition  to  those  which  may 
be  classified  as  acceptance  metrics  (Ref  52).  Code 
inspections  and  walk-throughs  are  oriented  towards  anomaly 
detection.  They  are  techniques  used  by  development  teams; 
yet  metrics  not  only  can  be  used  by  the  development  team, 
but  also  can  be  used  by  the  project  manager  as  acceptance 
criteria  (Ref  23). 

Identifying  Software  Quality  Requirements 

The  vehicle  for  establishing  software  quality 
requirements  is  the  hierarchical  model  of  software  quality 
(see  Figure  5,  Chapter  I)  defined  in  "A  Framework  for  the 
Measurement  of  Software  Quality",  Proceedings  of  the  ACM 
Software  Quality  Assurance  Workshop  in 
November  1978  (Ref  18). 

The  procedures  for  establishing  the  quality 
requirements  for  a  particular  software  system  utilize  this 
model  and  will  be  described  as  a  three  level  approach, 
which  are  the  levels  corresponding  to  the  hierarchical 
levels  of  the  software  quality  model.  The  basic  tool  to  be 
used  in  identifying  the  important  quality  factors  will  be 
the  Software  Quality  Requirements  Survey  form  shown  in 


Table  III.  The  formal  definitions  of  each  of  the  eleven 
quality  factors  are  provided  on  that  form  (Ref  17). 

It  is  recommended  that  a  briefing  be  provided  to  the 
decision  makers  using  the  tables  and  figures,  which  will  be 
described  later,  to  solicit  their  responses  to  the  survey. 
In  order  to  determine  the  quality  factors,  the  decision 
makers  should  consider  the  system  characteristics  as  shown 
in  Table  IV  (Ref  23).  In  other  words,  if  the  software 
system  is  to  have  a  long  life  cycle,  the  Quality  Factors 
that  must  be  considered  are  Maintainability,  Flexibility, 
and  Portability. 

Table  V  shows  that  the  eleven  quality  factors,  which 
are  identified  on  the  survey,  can  be  grouped  according  to 
three  life  cycle  activities  associated  with  a  delivered 
software  product.  These  three  activities  are  product 
operation,  product  revision,  and  product 
transition  (Ref  23).  The  cost  to  implement  versus  life 
cycle  cost  reduction  relationship  exists  for  each  quality 
factor.  The  benefit  versus  cost-to-provide  ratio  for  each 
factor  is  rated  as  high,  medium,  or  low  in  the  right  hand 
column  of  Table  V.  This  relationship  and  the  life  cycle 
implications  of  the  quality  factors  should  be  considered 
when  selecting  the  important  factors  for  a  specific  system. 

By  this  point,  a  tentative  list  of  quality  factors 
should  be  produced.  The  next  step  is  to  consider  the 
interrelationships  among  the  factors  selected.  Tal  le  VI 
and  Table  VII  can  be  used  as  a  guide  for  determining  the 
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Table  III 


Software  Quality  Requirements  Survey  Form 


The  11  quality  factors  listed  below  have  been  Isolated  from  the  current 
literature.  They  are  not  meant  to  be  exhaustive,  but  to  reflect  what  is 
currently  thought  to  be  important.  Please  indicate  whether  you  consider 
each  factor  to  be  Very  Important  (VI),  Important  (I),  Somewhat  Important  (SI), 
or  Not  Important  (NI)  as  design  goals  in  the  system  you  are  currently  working 
on. 


RESPONSE  FACTORS 


coahectsess 

ttnaaiiiTr 

gnciPCT 

tWTIWITV 

usability 

HAtHTMMatUTY 

TtSTABIUTY 

gpiBIUTT 

»o*-abihty 

AtUSABIUTY 

IXTtRQAWBtUTY 


DEFINITION 

Extent  to  which  a  propraa  satisfies  It  specifica¬ 
tions  and  fulfllli  the  user's  mission  objectives 

Extent  to  which  4  program  c jn  be  txptcttd  to  P*r- 
*orw  its  Intended  function  with  reoutrpd  precision. 

fh«  Mount  of  computlnp  resources  and  cod* 
requlreo  by  •  propraa  to  pqrfora  4  function. 

Extant  to  which  4CCPS1  to  software  a”  44t«  by 
unauthorised  parsons  can  b*  control  ltd. 

Effort  r«qul rod  to  loom.  ODOrat*.  propor*  Input, 
and  tntprpret  output  of  4  propraa. 

Effort  require*  to  locatt  and  fix  an  error  In  an 
Optra t Iona  I  propraa. 

Effort  raoulrcd  to  tost  a  propraa  to  Insure  It 
porfsms  Its  Intend**  function. 

Effort  required  to  atdtfy  an  operational  propria. 

Effort  rtqul  rod  to  transfer  a  pmpraa  froa  on* 
ha rm>*r*  configuration  and/or  software  syston 
onvlronaont  to  another. 

Extant  to  which  a  Pmpraa  can  b*  isod  In  other 
applications  -  related  to  the  gackapinp  and 
scop*  of  the  functions  that  propramt  perform. 

Effort  raoutrtd  to  coeol*  on*  systae  with 
another. 


2.  What  type(s)  of  application  are  you  currently  involved  in? 


3.  Are  you  currently  in: 

1.  Development  phase 

_ _ 2.  Operations/Maintenance  phase 

4.  Please  indicate  the  title  which  most  closely  describes  your  position. 

_ _  1.  Program  Manager 

_ 2.  Technical  Consultant 

_  3.  Systems  Analyst 

_ 4.  Other  (please  specify) _ 
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System  Characteristics  and  Related  Quality  Factors 


CHARACTERISTICS 


QUALITY  FACTORS 


If  human  lives  are  affected 


Reliabilit 


y 


t 


Long  life  cycle 


Maintainability 
F lexibility 
Portability 


Real  time  application 


E  ff ic ienc  y 

Reliability 

Correctness 


On-board  computer 
application 


E  fficiency 

Reliability 

Correctness 


Processes  classified 
information 


Integr  ity 


Interrelated  systems 


I nter operability 


I 


LEGEND:  A  -  where  quality  factors  should  be  measured  X  -  where  Impact  of  poor  quality  is  realized 


Table  ▼! 


LEGEND 


If  a  high  degree  of  quality  is  present  for  factor, 
what  degree  of  quality  is  expected  for  the  other: 

O  *  High  ^  =  Low 

Blank  =  No  relationship  or  application  dependent 


Table  VII 


SOME  TRADEOFFS  BETWEEN 
SOFTWARE  QUALITY  CHARACTERISTICS 


INTEGRITY  VS  EFFICIENCY— The  additional  code  and  processing  required 
to  control  the  access  of  the  software  or  data  usually  lengthen  run 
time  and  require  additional  storage. 

USABILITY  VS  EFFICIENCY— The  additional  code  and  processing  required 
to  ease  an  operator's  task  or  provide  more  usable  output  usually 
lengthen  run  time  and  require  additional  storage. 

MAINTAINABILITY  VS  EFFICIENCY— Optimized  code,  incorporating  intricate 
coding  techniques  and  direct  code,  always  provides  problems  to  the 
maintalner.  Using  modular,  instrumented,  and  wel 1- commented  high 
level  code  to  Increase  the  maintainability  of  a  system  usually 
Increases  overhead,  resulting  In  less  efficient  operation. 

TESTABILITY  VS  EFFICIENCY- -The  above  discussion  applies  to  testing. 

PORTABILITY  VS  EFFICIENCY— The  use  of  direct  code  or  optimized  soft- 
ware  or  utilities  decreases  the  portability  of  the  system. 

FLEXIBILITY  VS  EFFICIENCY— The  generality  required  for  a  flexible 
system  increases  overhead  and  decreases  the  efficiency  of  the  system. 

REUSABILITY  VS  EFFICIENCY— The  above  discussion  applies  to 
reusability. 

INTEROPERABILITY  VS  EFFICIENCY— Again  the  added  overhead  for  con- 
version  from  standard  protocol  and  standard  data  representations, 
and  the  use  of  interface  modules  decreases  the  operating  efficiency 
of  the  system. 

FLEXIBILITY  VS  INTEGRITY—Flexibi  1  i ty  requires  very  general  and 
flexible  data  structures.  This  increases  the  data  security  problem. 

REUSABILITY  VS  INTEGRITY— As  in  the  above  discussion,  the  generality 
required  by  reusable  software  provides  severe  protection  problems. 

INTEROPERABILITY  VS  INTEGRITY— Coupled  systems  allow  for  more 
avenues  of  access  and  different  users  who  can  access  the  system,  fho 
potential  for  accidental  access  of  sensitive  data  is  increased  as 
well  as  the  opportunities  for  deliberate  access.  Often,  coupled 
systems  share  data  or  software  which  compounds  the  security 
problems  as  well. 
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relationships  between  the  quality  factors  (Ref  28).  Some 
factors  are  synergistic  while  others  conflict.  The  impact 
on  conflicting  factors  is  that  the  cost  to  implement  will 
increase.  This  will  lower  the  benefit-to-cost  ratio.  For 
example,  there  is  a  very  low  relationship  between 
maintainability  and  efficiency.  It  would  be  unrealistic  or 
very  costly  for  a  user  to  expect  a  software  system  to  be 
highly  maintainable  and  highly  efficient.  An  efficient 
system  uses  optimized  code  and  incorporates  intricate 
coding  techniques;  as  such,  it  will  always  provide  problems 
to  the  maintainer.  On  the  other  hand,  a  highly 
maintainable  system  uses  modular  and  we 11 -commented  high 
level  code  which  increases  the  system  overhead,  resulting 
in  less  efficient  operation. 

Now  a  list  of  quality  factors  that  are  considered  to  be 
important  for  one  particular  system  should  be  compiled. 
The  list  should  be  organized  in  order  of  importance.  The 
definitions  of  the  factors  chosen  should  be  included  with 
this  last  list.  The  rationale  for  the  selection  of  factors 
must  be  documented. 

The  next  level  of  identifying  the  quality  requirements 
involves  proceeding  from  the  user-oriented  quality  factors 
to  the  software-oriented  criteria.  Sets  of  criteria,  which 
are  attributes  of  the  software,  are  related  to  the  various 
factors  by  definition.  Their  identification  is  automatic 
and  represents  a  more  detailed  specification  of  the  quality 
requirements.  Table  VIII  should  be  used  to  identify  the 
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software  attributes  associated  with  the 


chosen  critical 


quality  factors  (Ref  17).  For  exan£>le,  if  the  desired 
management -or ien ted  quality  factors  of  the  system  are 
Reliability  and  Maintainability,  then  the  software  must 
exhibit  the  following  criteria:  Consistency  and  Simplicity 
(both  common  to  Reliability  and  Maintainability)  as  well  as 
Accuracy  and  Error  Tolerance  (attributes  of  Reliability), 
and  Conciseness,  Modularity  and  Self-Descriptiveness 
(attributes  of  Maintainability). 

The  last  level,  which  is  the  most  detailed  and 
quantified,  requires  precise  statements  of  the  level  of 
quality  that  will  be  acceptable  for  the  software  product. 

Currently,  the  underlying  mathematical  relationships 
that  allow  measurement  at  this  level  of  precision  do  not 
exist  for  all  the  quality  factors.  The  relationships  exist 
for  reliability,  maintainability,  portability,  and 
flexibility  (Ref  28).  The  mechanism  for  making  the  precise 
statement  for  any  quality  factor  is  a  rating  of  the  factor. 
The  underlying  basis  for  the  ratings  is  the  effort  or  costs 
required  to  perform  a  function  such  as  to  correct  or  modify 
the  design  or  program  (Ref  23).  For  example,  rating  for 
maintainability  might  be  that  the  average  time  to  fix  a 
problem  should  be  five  man-days  or  that  90%  of  the  problem 
fixes  should  take  less  than  six  man-days.  This  rating 
would  be  specified  as  a  quality  requirement.  To  comply 
with  this  specification,  the  oftware  would  have  to  exhibit 
characteristics  which,  when  present,  give  an  indication 


that  the  software  will  perform  to  this  rating,  i.e.,  be 
fixed  in  less  than  six  man-days  if  a  problem  should  occur. 
These  characteristics  are  measured  by  metrics  which  are 
inserted  into  a  mathematical  relationship  to  obtain  the 
predicted  rating. 

In  order  to  choose  ratings  such  as  the  two  mentioned 
above,  data  must  be  available  which  allows  the  decision 
maker  to  know  what  is  a  "good  rating"  or  perhaps  what  is 
the  industry  average.  Currently  there  is  generally  a  lack 
of  good  historical  data  to  establish  these  expected  levels 
of  operations  and  maintenance  performance  for  software. 

There  are  significant  efforts  underway  to  compile 
historical  data  and  derive  the  associated  performance 
statistics  (Ref  9).  Individual  software  development 
organizations  should  attempt  to  compile  historical  data  for 
their  particular  environment. 

The  Automated  Measurement  Tool  (AMT),  which  is 
discussed  in  Chapter  IV,  has  been  created  to  collect  this 
data.  It  is  designed  to  be  both  language  and  machine 
independent  (Ref  18). 

The  vehicle  for  applying  the  software  quality  metrics 
are  the  metric  worksheets  contained  at  the  end  of  this 
report  in  Appendix  A.  Figure  6  illustrates  the  timing  of 
when  each  worksheet  is  to  be  applied  during  the 
SDLC  (Ref  23).  The  procedure  is  to  take  the  available 
documentation/source  code,  apply  the  appropriate  worksheet, 
and  translate  the  measurements  to  metric  scores.  For 
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DEVELOPMENT  PHASES 


REQUIREMENTS 

ANALYSIS 

DESIGN 

IMPLEMENTATION 

TEST 

AND 

INTEGRATION 

REQUIREMENTS 

SPEC 


METRIC 

WORKSHEET 

#  I 


PRELIMINARY 

DESIGN 

SPEC 

USER'S  MANUAL 
(DRAFT' 


METRIC 
WORKSHEET 
#  2a 


DETAILED 

DESIGN 

SPEC 

(BUILD  TO) 

TEST 

PLAN 

AND 

PROCEDURES 


METRIC 

WORKSHEET 

#  2b 

METRIC 

WORKSHEET 

#  2a 
UPDATE 


SOURCE 

CODE 

DETAILED 

DESIGN 

SPEC 

(BUILT  TO) 


TEST 

RESULTS 

USER'S  MANUAL 
(FINAL) 


METRIC  WORKSHEET 

#  3 

METRIC  WORKSHEET 

*  2b  UPDATE 


METRIC  WORKSHEET 
#  2a  UPDATE 


Figure  6. 

Application  of  the  Metric  Worksheets 


A 


example,  worksheet  #2a  would  be  used  during  the  design 
phase  of  a  system  development  effort.  This  worksheet 
should  be  able  to  collect  the  needed  system  level 
information  from  the  Preliminary  Design  Specifications  and 
the  Draft  of  the  Users  Manual.  Then  the  AMT  would  be 

< 

updated  with  this  information,  and  then  automatically 
compute  the  metric  scores. 

Techniques  for  Applying  Metrics 

The  metrics  can  be  applied  different  ways.  The  first 
technique  for  applying  the  metrics  is  by  formal  inspection. 

The  formal  inspection  is  performed  by  personnel  of  an 
organization  independent  of  the  development  organization 
(the  acquisition  office,  an  independent  quality  assurance 
group,  or  an  independent  contractor).  The  worksheets  are 
applied  to  delivered  products  at  scheduled  times  and  the 
results  are  formally  reported. 

The  second  technique  is  to  utilize  the  worksheets 
during  structured  design  and  code  walk-throughs  held  by  the 
development  team.  A  specific  participant  of  the 
walk-through  can  be  designated  for  applying  the  worksheets 
and  reporting  any  deficiencies.  During  the  walk-through  a 
representative  of  the  quality  assurance  organization  can 
participate  in  the  walk-through  with  the  purpose  of  taking 
the  measurements  of  the  design  or  code. 

The  last  technique  is  for  the  development  team  to  use 
the  worksheets  as  guidelines,  self-evaluations  or  in  a  peer 
review  mode  to  enhance  the  quality  of  the  products  they 
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produce. 

These  Software  Quality  Metrics  are  now  sufficiently 
developed  to  use  within  a  SQA  program.  The  AMT  reduces  the 
requirement  for  human  resources  by  automating  much  of  the 
collection  of  data. 

Chapter  III  presents  the  methods  used  to  derive  the 
information  needed  to  make  recommendations  concerning  the 
development  of  AFLC/LM's  Software  Quality  Assurance 
Program.  It  discusses  the  research  required  to  demonstrate 
the  feasibility  of  incorporating  Software  Quality  Metrics 
into  APLC's  SQA  program. 
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CHAPTER  III 


RESEARCH  METHODOLOGY 


The  overall  objective  of  this  thesis  is  to  provide 
guidelines  to  AFLC/LM  concerning  the  way  Software  Quality 
Metrics  should  be  integrated  into  its  Software  Quality 
Assurance  program.  This  chapter  will  discuss  the 
methodology  used  to  attain  this  objective  and  indicate  how 
the  four  research  goals  specified  in  Chapter  I  were 
achieved. 

Establishing  the  Framework 

In  order  to  accomplish  the  first  goal:  To  determine 
which  measurement  tools  and  techniques  should  be  used  to 
support  a  SQA  program,  a  framework  for  analysis  was 
established  by  reviewing  current  Software  Quality  Assurance 
programs  throughout  DoD  and  the  industry.  This  provided  a 
baseline  to  determine  how  software  quality  has  been 
assessed. 

Since  it  was  critical  to  have  a  broad  source  of  current 
information,  tutorials  and  proceedings  were  read  from 
conferences  and  symposiums  on  Quality  Management,  Quality 
Control,  Quality  Assurance,  Software  Design,  Software 
Management,  Software  Engineering,  and  Configuration 
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Management.  Some  of  the  more  significant  contributions  in 
this  survey  were:  Donald  J.  rteifar's  "Tutorial  on  Software 
Management",  an  IEEE  Catalog  in  1979  (Ref  7),  Freeman  and 
Wasserman's  "Tutorial  on  Software  Design  Techniques",  an 
IEEE  Catalog  in  1980  (Ref  8),  and  Bryan,  Chadbourne,  and 
Siegel's  "Tutorial  on  Software  Configuration  Management", 
an  IEEE  Catalog  in  1980  (Ref  50). 

The  tutorials  provided  between  thirty  to  fifty-five 
articles  each  on  current  topics  concerning  the  software 
system  development  life  cycle.  This  review  helped  provide 
an  overview  of  what  tools  and  techniques  were  being 
effectively  used  in  industry  and  the  government  in  building 
Software  Quality  Assurance  programs. 

Attendance  at  the  DPMA  National  Symposium  on  Effective 
Methods  of  EDP  Quality  Assurance  on  1-3  April  81  in 
Chicago,  IL,  provided  the  opportunity  to  speak  to  Canadian 
and  U.S.  Quality  Assurance  experts.  The  full  day  workshop 
on  "Establishing  the  Quality  Assurance  Function"  and  the 
two  day  conference  which  hosted  several  QA  experts  who 
presented  material  on  current  QA  efforts  and  directions 
provided  insights  to  recent  industrial  developments  in  QA. 

Membership  in  ACM  and  IEEE  allowed  attendance  at  their 
local  meetings  which  host  speakers  who  discuss  recent 
developments  in  software  and  hardware.  Additionally, 
membership  in  four  of  ACM's  Special  Interest  Groups: 
SIGSOFT,  SIGSIM,  SIGCAS,  and  SIGBDP  and  two  of  IEEE' s 
Societies:  Engineering  Management  aciety  and  Computer 
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Society  have  provided  all  their  publications  which  keep 
members  current  on  developments  and  practices  in  the 
computer  field. 

Membership  in  The  Library  of  Computer  and  Information 
Sciences  Book  Club,  provided  recent  publications  that  the 
libraries  in  this  area  do  not  have.  The  books  that  were 
purchased  cover  Quality  Control,  Project  Management, 
Software  Quality  Management,  Software  Engineering,  and 
Structured  Requirements  Definition. 

By  September  of  1981,  most  of  the  exploratory  research 
had  been  completed.  By  then  it  was  clear  that  Software 
Metrics  represented  the  "leading  edge"  of  SQA  because  it 
promised  the  ability  to  quantify  quality  by  providing 
objective  measurement  (Ref  140). 

Model  of  Development  Decision  Process 

The  second  goal:  To  develop  a  model  of  the  decision 
making  process  of  AFLC/LM's  software  system  development 
effort  which  incorporates  the  application  of  Software 
Quality  Metrics,  was  accomplished  in  three  phases. 

Phase  1.  During  this  phase,  interviews  and  additional 
literature  surveys  were  conducted  to  determine  how  current 
research  in  systems  modeling  and  software  metrics  might 
help  in  solving  AFLC's  problems  in  software  development. 

Interviews  with  AFLC/LM  personnel  were  conducted  to 
clearly  define  the  current  problems  in  software 
development,  and  to  discuss  how  software  metrics  might 
provide  some  of  the  solutions  by  providing  a  quality 
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assessment  capability.  Richard  Woodward,  the  AFLC/LM 
sponsor  of  this  thesis,  provided  the  initial  QA  background 
at  AFLC.  Cloyd  D.  Stratton  provided  information  about 
Configuration  Management  and  its  evolution  at  AFLC/LM. 
Jack  Tinsley  provided  information  which  described  how  CM  is 
functionally  carried  out,  i.e.,  "who  does  what"  during  the 
SDLC. 

Additional  literature  surveys  were  conducted  to 
determine  who  had  been  doing  the  research  in  software 
metrics  and  where  the  software  metrics  were  being  applied. 
This  required  a  review  of  DoD  material  on  SQA. 

Because  Rome  Air  Development  Center  (RADC )  was 
sponsoring  research  in  the  area  of  software  metrics,  a 
point  of  contact  was  established  with  Joe  Cavano  at  Griffis 
AFB,  NY.  Results  of  RAOC's  testing  were  not  available 
until  March  82,  at  which  time  they  were  requested. 

After  discovering  the  efforts  of  James  McCall  and 
others  at  General  Electric’s  Command  and  Information 
Systems  Division  at  Sunnyvale,  CA,  materials  were  obtained 
on  their  initial  findings  about  Software  Quality  Metrics. 
In  order  to  clarify  certain  issues  propose!  by  this 
documentation  concerning  Software  Quality  Metrics,  two  days 
were  spent  interviewing  James  McCall  and  Dave  Markham  at 
G.E.  on  October  29  and  30  to  determine  the  usefulness  of 
SQM  and  the  validity  of  the  metrics  that  they  had  derived. 
Since  G.E.'s  effort  to  derive  and  validate  the  Software 
Quality  Metrics  is  of  obvious  importance  to  the  usefulness 
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of  SQM  as  a  measurement  tool  in  a  SQA  program,  a  detailed 
discussion  of  G.E.'s  efforts  has  been  included  in  this 
thesis. 

Phase  2.  This  phase  involved  developing  a  model  which 
displays  the  interaction  between  the  software  development 
decision-making  process  and  the  user  environment  using  QA 
and  systems  concepts  acquired  in  six  AFIT  courses: 
EE  5.45 — Software  Systems  Acquisition,  EE  6.99B — Software 
Management  Information  Requirements,  EE  6.93 — Software 
Engineering,  SM  6.44 — Techniques  in  Project  Management, 
LM  6.15 — Logistics  Decision  Support  Systems,  and  MA 
5.48 — Computer  Systems  Analysis. 

The  model  represents  the  SDLC  decisioning  process.  It 
illustrates  the  interaction  between  the  software 
development  management  organization  (Project  Management 
including  the  SQA  function)  and  the  user  environment. 
Moreover,  each  subsystem  within  this  moael  (scanning, 
intelligence,  and  decision)  is  displayed  in  a  manner  that 
clearly  demonstrates  the  interrelationships  between  its 
input,  output,  process,  and  feedback  mechanisms. 

Phase  3.  Finally,  in  order  to  superimpose  the  Software 
Quality  Metrics  framework  onto  the  Phase  2  conceptual 
model,  the  procedures  developed  by  General  Electric  for 
applying  the  metrics  were  "fit"  into  the  decision-making 
process  to  reveal  how  the  metrics,  by  providing  quantified 
measures  of  software  quality  during  each  phase  of  the  SDLC, 
can  improve  the  end-product  of  the  software  development. 
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Metric  Application 

The  third  goal:  To  demonstrate  how  the  SQM  concepts 
can  be  applied  on  an  existing  system  at  AFLC ,  was  achieved 
by  the  development  of  checklists  which  are  to  be  used  by 
system  developers.  Documentation  was  acquired  through 
Robert  Brooks,  AFLC/LMX,  who  is  the  Project  Coordinator  for 
the  Maintenance  Management  Systems  Improvement  Project 
(MMSIP).  Through  Mr.  Brooks,  the  documentation  on  G072A 
MMSIP,  which  is  a  subsystem  of  MMSIP,  was 
obtained  (Ref  142). 

The  procedures  and  metrics  worksheets,  which  were 
developed  by  General  Electric,  were  applied  to  this 
subsystem.  Because  the  subsystem  had  just  become 
operational,  the  development  documentation,  which  is 
described  in  Appendix  F,  was  still  available. 

To  satisfy  the  request  of  the  sponsor,  AFLC/LM,  the 


real  intent 

of  this 

third 

goal 

was 

to  indicate  what 

changes,  if 

any,  had 

to  be 

made  to 

current  documentation 

requirements 

in  order 

to 

utilize 

the 

Software  Quality 

Metrics  in  a  Software  Quality  Assurance  program  for 
AI'LC/LM.  Therefore,  checklists,  which  account  for  the 
software  —or ien ted  requirements  for  Reliability  and 
Maintainability,  were  developed  to  insure  that  software 
developers  know  in  advance  that  specific  items  should  be 
ccnsidered  in  developing  reliable  and  maintainable 
software . 
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By  tu  lfilling  this  request,  AFLC/LM  will  be  able  to 
acquire  the  Automated  Measurement  Tool  (AMT)  from  Rome  Air 
Development  Center  and  already  be  set-up  to  use  the  tool  in 
their  SQA  programs. 

Metric  Integration 

The  achievement  of  the  fourth  goal:  To  use  the 
information  generated  from  this  research  to  help  AFLC/LM 
formulate  guidelines  to  integrate  Software  Quality  Metrics 
and  other  software  tools  into  a  Software  Quality  Assurance 
program,  allowed  recommendations  to  be  made  concerning : 

1.  what  changes  in  current  Configuration  Management 
practices,  such  as  in  documentation  of  the  system 
development  life  cycle,  should  be  made  to  meet  the 
information  requirements  of  measurement  tools; 

2.  what  software  tools  should  be  acquired  to  support 
the  collection  of  SQM; 

3.  to  what  extent  the  Software  Quality  Metric 
framework  could  be  used  if  the  automated  tools  are  not 
acquired;  and  finally 

4.  what  further  applications  can  be  seen  for  the  use 
of  metrics. 

Chapter  IV  accomplishes  the  first  goal  of  this  thesis 
by  reviewing  current  research  in  Software  Quality  Assurance 
(SQA)  and  analyzing  software  measurement  tools  and 
techniques. 


50 


CHAPTER  IV 


SQA  TOOLS  AND  TECHNIQUES:  A  SURVEY  OF  DOD  AND  INDUSTRY 


A  Review  of  current  Software  Quality  Assurance  (SQA) 
programs  throughout  DoD  and  the  industry  has  provided  a 
baseline  to  determine  how  software  quality  has  been 
assessed.  The  initial  literature  search  revealed  a 
diversified  description  of  software  attributes.  Table  IX 
summarizes  the  list  of  software  attributes  and  indicates 
the  literature  source  for  each  attribute.  Appendix  E 
provides  the  various  definitions  for  each  software 
attribute,  as  it  is  described  in  the  literature. 

This  initial  survey  demonstrated  the  need  for  a 
diversified  set  of  SQA  tools  and  techniques  that  would  be 
required  to  assess  the  characteristics  of  a  software 
system.  Related  research  by  General  Electric  has  grouped 
these  software  attributes  into  eleven  software  factors, 
with  the  related  attributes  serving  as  criteria  which 
further  define  each  factor.  This  consolidation  of  software 
characteristics  is  supported  by  other  research  which  has 
used  these  factors  to  make  comparisons  of  SQA  tools  and 
techniques  (Refs  15,  23). 

The  goal  of  this  chapter  is  to  determine  which 

measurement  tools  and  techniques  should  be  used  to  support 


Table  IX 


HI 


a  SQA  program  for  AFLC/LM.  To  accomplish  this  goal, 
categories  for  software  QA  tools  and  techniques  (aids)  have 
been  presented.  Next,  material  on  "toolsmithing"  has  been 
provided,  and  an  assessment  of  state-of-the-art  technology 
in  SQA  aids  has  been  made.  Finally,  a  discussion  about  the 
use  of  Software  Quality  Metrics  (SQM)  and  the  Automated 
Measurement  Tool  (AMT)  concludes  the  chapter. 

Techniques,  Tools,  and  Methodologies 

Experience  has  indicated  that  good  techniques,  tools, 
and  methods  must  be  employed  to  have  a  successful  quality 
assurance  program.  Software  QA  is  a  service  function 
created  to  provide  management  with  the  independent  checks 
and  balances  it  needs  to  control  and  assess  the  software 
system  quality.  Economic  considerations  are  such  that  the 
SQA  function  must  accomplish  its  assigned  task  efficiently 
and  at  minimum  cost  to  remain  effective.  Technical 
considerations  are  such  that  SQA  can  only  justify  its  role 
if  it  contributes  directly  to  the  system  development  with 
minimal  disruption  and  meaningful  results.  Both  economic 
and  technical  considerations  dictate  that  SQA  use  proven 
tools  to  automate  the  technjques  and  methods  it  employs  to 
accomplish  its  purpose  (Ref  )5). 

For  the  purpose  of  this  thesis,  automation  is  defined 
as  mechanizing  methods  using  tools  and  techniques.  Tools 
are  computer  programs  used  to  aid  the  quality  inspector  in 
evaluation  of  the  developer's  software  system.  Techniques 
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are  procedures  arranged  to  simplify  the  evaluation 
process  (Ref  15). 

As  shown  in  Table  X,  SQA  Tools  and  Techniques,  software 
QA  personnel  employ  the  methods  of  inspection,  analysis, 
demonstration,  and  test.  Inspection  confirms  product  or 
procedure  compliance  with  stated  requirements  by 
examination.  Analysis  studies  the  product  or  procedure  in 
detail  in  order  to  analytically  confirm  an  answer  or 
results  of  a  solution.  Demonstration  provides  tangible  and 
visible  evidence  of  compliance  by  making  trial  output 
acceptable  for  review  and  comparison  against  stated  goals. 

Software  QA  tools  and  techniques  can  be  classified 
based  upon  the  method  they  support.  Table  X  lists 
available  tools  and  techniques  by  category  or  class. 
Appendix  H  provides  a  glossary  for  these  SQA  tools  and 
techniques.  It  should  be  noted  that  only  the  Software 
Quality  Metrics  techniques  and  the  Automated  Measurement 
Tool  (AMT)  support  the  full  spectrum  of  SQA  methods. 

Techniques.  Table  XI  illustrates  which  techniques 
support  the  SQA  functions.  These  functions  are 
representative  of  QA  functions  that  are  specified  in 
MIL-S-52779A,  Software  Quality  Assurance  Program 
Requirements  (Ref  86).  As  shown  in  the  table,  Software 
Quality  Metrics,  Reviewing,  Auditing,  and  Standardization 
all  support  the  full  range  of  SQA  functions.  AFLC/LM 
incorporates  reviews  and  audits  throughout  the  software 
development  life  cycle  (SDLC)  under  Configuration 


Supporting  Techniques  for  SQK  Functions 


13.  Stress  testing 

20.  Symbolic  execu 

21.  Walic-throughs 


Management  (CM),  and  it  is  striving  to  improve  its 
procedures  for  standardization  (Ref  38).  Software  Quality 
Metrics  (SQM)  provides  the  procedures  which  incorporate 
Reviewing,  Auditing,  and  Standardization,  and  it 
establishes  guidelines,  in  the  form  of  checklists,  which 
enhance  other  SQA  techniques  in  use  (Ref  23). 

Table  XII  reveals  the  effectiveness  of  SQA  techniques 
in  assessing  quality  properties.  The  techniques  that 
possess  the  highest  assessment  capability  for  all  the 
quality  factors  are:  Design  Inspection,  Software  Quality 
Metrics,  and  Standardization  (Walk-throughs  provide  "ease 
in  use"  as  well  as  effectiveness  in  assessment). 

Tools.  Table  XIII  depicts  the  tools  that  support 
various  SQA  functions  that  are  specified  in 

MIL-S-52779A  (Ref  86).  The  Automated  Measurement  Tool 
(AMT),  Standards  and  Text  Editors  are  the  only  tools  which 
support  the  full  spectrum  of  SQA  functions  (Refs  15,  31). 

Table  XIV  displays  the  effectiveness  each  tool  provides 
for  assessing  software  quality.  Only  the  AMT,  Standards, 
and  Test  Bed  tools  are  highly  effective  in  assessing  all 
quality  properties.  The  Language  Processor,  Standards 
Analyzer,  and  Dynamic  Analyzer  also  provide  effectiveness 
to  SQA  program  (Refs  15,  84). 

Toolsmithing 

As  can  be  ascertained  from  the  previous  section,  the 
software  quality  engineer  has  many  tools  and  techniques  at 
his  disposal.  One  of  the  quality  engineer's  major  tasks  in 
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Technique  Effectiveness  in  Assessing  Quality  Properties 


effectiveness  M  -  Medium  effectiveness  L  -  Low  effectiveness 


Tab  la  XIII 


Supporting  Tools  for  SQA  Functions 


Function 


Tools 

Quality  planning 

Work  tasking 

Configuration 

management 

Testing 

Corrective  action 

Library  controls 

Design  standards 

1.  Accuracy  study  processor 

X^ 

2.  Automated  Measurement  Tool 

X 

X 

X 

X 

X 

X 

X 

3.  Automated  test  generator 

X 

4.  Comparator 

X 

X 

5.  Consistency  checker 

X 

6.  Cross-reference 

X 

7.  Data  base  analyzer 

X 

8 .  Debugger 

X 

9.  Decision  tables 

X 

X 

X 

X 

X 

10.  Dynamic  analyzer 

X 

11.  Dynamic  simulator 

X 

12.  Editor 

X 

13.  Flowcharter 

X 

X 

14.  Hardware  monitor 

X 

15.  Instruction  trace 

X 

16.  Interface  checker 

X 

17.  Interrupt  analyzer 

X 

18.  Language  processor 

X 

X 

19.  Libraries 

X 

X 

20.  Logic  analyzer 

X 

21.  MIS 

X 

X 

X 

X 

X 

22.  Requirements  tracer 

X 

X 

X 

23.  Simulator 

X 

X 

24.  Software  monitor 

X 

25.  Standards 

X 

X 

X 

X 

X 

X 

X 

26.  Standards  analyzer 

X 

X 

27.  Static  analyzer 

X 

X 

28.  Structure  analyzer 

X 

X 

29.  Test  bed 

X 

30.  Test  drivers,  sc? ipts 

X 

31.  Test-result  processor 

X 

32.  Text  editor 

X 

X 

X 

X 

X 

X 

X 

33.  Timing  analyzer 

X 

X 

X 

X 


X  X 


X 

X 

X  X 

X  X 

X  X 


c  o 


Design  standards 


Table  xrr 

Teel  Effectiveness  in  Assessing  Quality  Properties 


Usability 


developing  an  acceptable  software  quality  assurance  program 
for  a  specific  organization  and/or  project  is  to  select 
those  tools  and  techniques  that  will  allow  him  to 
accomplish  his  functions  in  the  most  efficient  and 
cost-effective  manner.  Selection  assumes  that  the  quality 
assurance  organization  has  its  methods  and  techniques 
codified  in  written  form  (i.e.,  usually  as  procedures  in  a 
manual),  its  tools  developed  and  maintained  under 
configuration  control  in  a  centralized  library,  and  its 
sources  for  supplementing  existing  tools  and  techniques 
identified  in  case  acquisition  is  warranted  (Ref  15). 
Factors  which  impact  selection  and  should  be  evaluated  are 
listed  in  checklist  form  in  Table  XV  (Ref  87). 

This  checklist  should  be  used  to  evaluate  each  tool  and 
technique  that  is  considered  for  inclusion  in  a  SQA 
program.  Using  this  checklist  to  assess  Software  Quality 
Metrics  (SQM)  and  the  Automated  Measurement  Tool  (AMT)  as 
possible  aids  in  the  SQA  program  for  AFLC/LM,  a  manager  or 
software  quality  engineer  will  quickly  realize  that  SQM  and 
AMT  should  be  acquired  because  every  response  on  the 
checklist  is  affirmative: 

Applicability.  The  AMT  and  SQM  are  suited  for  the  task. 

They  provide  the  needed  measures,  and  already  check 
the  quality  of  COBOL  source  code  (Ref  84). 

Availability.  The  SQM  and  AMT  are  now  ready  for  use. 

Rome  Air  Development  Center  completed  testing  and 
validation  in  March  1982.  The  AMT  was  prototyped  on 
the  VAX  11/780  and  modified  to  be  used  by  the  RADC 
H6 180  (Ref  84). 

Cost/Benefit.  The  software  and  tapes  for  the  AMT  and  the 
supporting  documents  for  SQM  can  now  be  acquired  by 
DoD  agencies  by  submitting  the  request  to  RADC. 
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actors  Impacting  Tool  and  Technique  Selection 


risks  associated  with  developing  or  using 
ir  technique  been  quantified? 


Experience.  The  SQM  and  AMT  have  been  used  on  both  Army 
and  Air  Force  projects.  The  documentation  provides 
the  strengths  and  current  limitations  (Ref  23). 

Quality.  The  SQM  and  AMT  are  we 11 -documented ,  are  under 
configuration  control,  and  have  been  qualified  through 
DoD  contracts.  The  inqplementation  of  the  AMT  was  done 
in  IFTRAN,  a  General  Research  Corporation  structured 
programming  preprocessor  to  FORTRAN  (Ref  89). 

Resources.  The  SQM  and  AMT  documentation  provide  the 
information  needed  for  conversion  to  the  AFLC 
environment.  However,  further  expansion  of  metric 
application  could  be  achieved  with  development 
assistance  by  G.E. 

Risk.  The  only  risk  involved  in  using  SQM  and  the  AMT  is 
if  AFLC  does  not  first  develop  a  database  of  opera¬ 
tional  history  of  its  systems  before  relying  on  the 
predictive  accuracy  of  the  metric  scores.  The  AMT  can 
be  used  to  build  this  data  base  (Ref  51). 

Based  on  the  previous  discussion  of  SQA  aids,  it  is 

believed  that  the  tools  and  techniques  displayed  in 

Table  XVI  represent  the  minimum  set  required  to  construct  a 

responsive  and  quantitative  SQA  program  (Refs  15,  24). 

This  set  provides  for  a  balanced  coverage  of  the  required 

SQA  functions  as  illustrated  in  Tables  XI  and  XTll.  it 

also  provides  for  a  balanced  assessment  of  the  software 

quality  factors  as  illustrated  in  Tables  XIT  and  XIV. 


Table  XVI 

Hiniaua  Set  of  Tools  and  Techniques 


Other  tools  and  techniques  may  be  added  as  the  need 
arises,  so  long  as  their  cost  can  be  justified  in  terms  of 
benefits  derived.  The  quantitative  aspect  of  this  minimum 
set  of  tools  and  techniques  is  a  direct  result  of  the 
inclusion  of  SQM  and  the  MIT. 

Chapter  II  provides  the  overview  for  using  SQM; 
Chapter  V  presents  a  detailed  application  of  the  SQM 
concepts  throughout  the  SDLC,  and  Appendix  J  discusses  the 
procedures  for  quantitatively  assessing  software  quality  by 
using  the  lowest  level  of  SQM — the  metrics.  However,  from 
a  management  perspective  of  the  SDLC,  it  is  the  automated 
measurement  services,  which  operationalize  the  metric 
concepts,  that  should  prompt  AFLC/LM  to  acquire  the  AMT. 

Automated  Measurement  Services 

The  concept  behind  SQM  is  to  provide  software  managers 
with  a  mechanism  to  quantitatively  specify  and  measure  the 
level  of  quality  in  a  software  system.  To  provide  this 
mechanism,  a  software  developer  must  collect  data  from  the 
products  of  the  software  development  process.  This  raw 
data  is  then  used  to  calculate  metric  values  which  can  be 
used  to  assess  the  quality  of  the  software  as  it  is  being 
produced.  The  purpose  of  the  A  itomated  Measurement  Tool 
(AMT)  is  to  provide  automated  support  to  the  application  of 
the  SQM  concepts.  Without  the  AMT,  the  metrics  data  must 
be  collected  by  hand  in  a  tedious,  error-prone,  and 


time-consuming  process  (Ret  84). 

The  amount  of  data  that  can  be  automatically  collected 


by  the  AMT  is  limited  to  data  that  can  be  derived  from 
machine  readable  sources  such  as  design  materials  generated 
on  the  computer  and  the  actual  source  code.  The  current 
version  of  the  AMT  automatically  collects  and  stores  raw 
data  from  COBOL  source  code  (most  of  AFLC/LM's  software  is 
written  in  COBOL).  The  remainder  of  the  raw  data  required 
to  calculate  the  metrics  must  be  manually  collected. 

A  SQA  analyst  uses  the  worksheets  in  Appendix  A  to 
answer  questions  about  the  system  being  measured.  When  the 
worksheet  is  complete,  its  data  is  entered  into  the  AMT 
database,  where  it  is  stored  along  with  the  automatically 
collected  data. 

Currently  the  AMT  collects  data  from  COBOL  source  code 
for  individual  modules.  A  total  of  25  different 
measurements  are  collected  automatically.  These 

measurements  can  be  divided  into  the  following  broad 
classes : 

1.  Counts  of  total  number  of  code  statements  and 
comment  statements; 

2.  Counts  for  individual  types  of  statements,  i.e., 
input,  output,  exit,  entry,  declarative,  etc. ; 

3.  Counts  of  different  types  of  branching  statements 
both  conditional  and  unconditional,  and 

4.  Counts  of  operands  and  operators,  for  use  in  cal¬ 
culating  Halstead's  measure  (Ref  23). 

The  specific  data  items  automatically  collected  are  shown 

in  Table  XVII  (Ref  84).  Also  noted  in  the  table  is  the 

entry  on  the  worksheet  that  the  item  relates  to  and  the 

metric  that  it  helps  calculate.  The  worksheet  entry  is 

identified  by  worksheet  number  (WS#),  section  number  ( S # ) , 

and  item  number  (I#).  Thus  a  notation  such  as  WS 3 ,  SI,  II 
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is  read  as  worksheet  3,  section  I,  item  1. 


This 

automated 

support  counts 

for 

25 

of  the  92 

worksheet 

#3  data 

items,  or  27% 

of 

the 

measurements 

required  at  the  implementation  phase  of  a  software 
development.  These  25  data  items  help  calculate  9  of  the 
38  metrics  related  to  implementation,  or  24%  of  those 
metrics.  The  metric  calculation  is  described  later  under 
Report  Generation  Services  (Ref  17). 

The  automated  data  collection  is  performed  by  the 
Automated  Measurement  Services  (AMS)  and  the  Preprocessing 
Service  (PPS)  subsystems.  The  user  invokes  these 
subsystems  using  the  MEASURE  command.  The  Preprocessing 
Services  uses  an  >1)  generalized  parser  to  decompose  the 
COBOL  source  code  cor.  cal  nod  in  the  file  identified  by  the 
MEASURE  command.  The  result  of  the  parsing  is  a  parsec 
tree  representation  of  the  source  code.  The  AMS  subsystem 
then  traverses  the  parse  tree  and  counts  the  various  data 
items  and  enters  them  in  the  data  base. 

The  significant  aspect  of  this  approach  is  that  other 
language  grammars  can  be  described  to  the  parser,  a  scanner 
developed,  and  the  parser  will  produce  a  parse  tree 
representation  of  the  other  language.  With  careful 
attention  paid  to  the  token  representation,  the  AMS  will  oe 
able  to  process  this  parse  tree  representation  of  the  other 
language  with  no  modif ication. 

In  addition  to  it  being  language  independent,  the  AMT 
was  developed  with  the  concept  of  eventually  interfacing  it 
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to  other  software  tools  in  a  software  development 
environment.  The  interfacing  would  be  done  by  extracting 
metric  data  available  from  the  processing  done  by  the  other 
tools  and  inserting  the  data  into  the  AMT  database  so  that 
metrics  could  be  calculated. 

A  program  would  have  to  be  written  which  extracts  the 
appropriate  data  from  the  output  file  of  a  tool,  and  using 
the  AMT  PUT  command,  to  then  insert  the  data  into  the 
database.  Potential  tools  that  should  be  considered  are: 
Requirements  Specification  Language  processors/analyzers. 
Program  Design  Language  processors/analyzers,  code 
auditors,  code  instrumentors,  and  Configuration  Management 
tools.  Appendix  1  provides  a  tool  survey  of  specific  SQA 
tools  that  could  be  considered  (Ref  84). 

Report  Generation  Services 

The  Report  Generation  Services  (RGS )  provides  the  user 
the  ability  to  generate  various  reports  that  reflect  the 
contents  of  a  database.  Nine  reports  may  be  requested  by 
the  user  to  display  the  metric  data  in  a  variety  of 
formats,  and  by  performing  additional  calculations,  present 
various  forms  of  data  both  at  the  module  and  system  levels. 
Basically,  data  is  extracted  from  the  database  to  calculate 
metric  values.  The  algorithms  for  performing  these 
calculations,  which  are  listed  in  Appendix  D,  are  contained 
in  the  RGS  routines.  Samples  of  the  AMT  reports  are  found 
in  Appendix  K,  and  a  brief  description  of  each  report 
follows  (Ref  84): 
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MODULE  REPORT.  This  report  displays  the  catalog  of 
modules  that  have  been  entered  into  the  database.  It 
provides  a  status  report  on  the  database. 

METRIC  REPORT.  This  report  calculates  the  value  of 
each  metric  categorized  by  factor  and  by  development  phase. 
This  report  is  used  to  determine  a  total  picture  of  the 
project  as  measurements  are  taken. 

EXCEPTION  REPORT.  The  exception  report  delivers  the 
relationship  of  each  module  to  a  given  threshold  value  of  a 
particular  metric.  The  relationship  (less  than,  equal  to, 
or  greater  than)  and  the  threshold  value  is  input  from  the 
user.  This  report  can  be  used  to  identify  modules  whose 
scores  do  not  meet  a  certain  threshold,  identifying  them  at: 
potential  problems. 

QUALITY  GROWTH  REPORT.  When  the  user  wishes  tc  track 
the  value  of  a  particular  metric  over  time,  the  Quality 
Growth  Report  will  furnish  a  tabular  display  of  the  scores 
of  a  selected  metric  over  the  phases  of  the  project.  This 
report  is  used  to  track  a  particular  metric  through  a 
project  to  see  how  its  value  changes. 

NORMALIZATION  REPORT.  The  Normalization  Report 
provides  the  user  with  the  overall  rating  of  a  selected 
quality  factor.  A  series  of  regression  equations  are 
displayed  which  have  been  empirically  derived  from 
research.  The  current  metric  values  are  substituted  in  the 
equations  and  a  rating  for  the  selected  quality  factor  is 
calculated.  Regression  equations  exist  for  the  quality 
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factors  Reliability,  Maintainability,  Portability,  and 
Flexibility  only. 

STATISTICS  REPORT.  The  Statistics  Report  provides  a 
profile  of  COBOL  constructs  for  each  module. 

SUMMARY  REPORT.  The  summary  report  provides  a  summary 
of  the  metric  scores  for  all  of  the  modules  in  the  system. 

WORKSHEET  REPORT.  The  worksheet  report  displays  the 
raw  data  entered  in  each  worksheet.  It  represents  the 
current  values  in  the  database.  It  is  used  to  verify  and 
track  data  entry. 

MATRIX  REPORT.  This  report  displays  the  average  and 
standard  deviations  for  all  metrics  values  for  all  modules. 
This  report  displays  all  of  this  information  in  a  matrix 
form  allowing  the  user  to  easily  identify  modules  with 
metric  scores  that  vary  from  the  system  average. 

The  reports  may  be  classified  as  to  their  primary  use: 
Descriptive,  Historical,  and  Diagnostic.  The  reports  that 
are  descriptive  are  ‘vhe  Summary,  Matrix,  Module,  and  Metric 
reports.  Their  common  characteristic  is  that  they  report 
data  and  in  content  or  format  implying  no  judgements 
concerning  the  data.  The  Summary  Report  reports  all  metric 
scores  for  each  module  or  for  all  the  modules.  It  is  a 
detailed  and  relatively  long  report.  The  Matrix  Report 
displays  the  mean  and  standard  deviation  of  the  modules  for 
each  metric.  It  is  a  good  snapshot  of  the  data  in  the  data 
base.  The  Module  Report  is  meant  for  operational 
personnel.  It  simply  reports  those  names  of  the  modules 
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which  are  in  the  data  base.  The  Metric  Report  is  a  more 
detailed  output  which  displays  the  metric  values  for  each 
module  in  a  detailed  form. 

The  Historical  Reports  are  the  Quality  Growth  and 
Worksheet  Reports.  The  Quality  Growth  report  provides  the 
quality  trend  of  a  module  through  the  development  phases. 
The  Worksheet  Report  gives  a  very  detailed  display  of  the 
raw  data  before  it  is  transformed  into  metric  scores.  Its 
main  use  is  to  track  data  entry  and  updates. 

The  Diagnostic  Reports  are  those  that  show  potential 
problem  areas.  They  are  the  Normalization,  Exception,  and 
Statistics  Reports.  The  Normalization  Report  applies  the 
regression  equations  derived  from  research  to  metric  values 
related  to  the  quality  factors  of  flexibility, 
maintainability,  portability,  and  reliability.  These 
regression  equations  have  been  developed  through 
examination  of  previous  projects  and  correlating  their 
metric  data  with  various  time  and  cost  data  (Ref  23). 

Regression  equations  for  the  remaining  quality  factors 
have  not  been  established.  The  Exception  Report  provides  a 
comparison  of  the  metric  scores  with  predetermined,  user 
supplied  values.  The  Statistics  Report  gives  a  diagnostic 
snapshot  of  any  module.  These  data  may  be  used  to  evaluate 
standards  or  identify  potential  problem  areas  (Ref  84). 

Table  XVIII  indicates  the  reports  that  support  the 
decisions  and  actions  related  to  program  management, 
developers,  software  quality  assurance,  and  test.  Some 


T»bl«  XVIII.  Support  To  Personnel 


PERSONNEL 

SUPPORT 

REPORT  TYPE 

Program  Management 

Progress  with  Respect 

Normalization  Function 

to  Quality  Goals 

Quality  Growth  Metrics 

Developers 

Standards  Enforcement 

Metrics 

Design  Decisions 

Summary 

Quality  Assurance 

Standards  Enforcement 

Metrics 

Compliance  with  Quality 

Quality  Growth 

Goals  Problem  Identification 

Exception 

Normalization  Function 

Test 

Test  Strategy 

Metrics 

Test  Emphasis 

Summary 

Test  Effort 

Except  Ion 

2 


examples  are: 

The  Normalization  Report  calculates  an  overall  rating 
of  a  particular  quality  factor  such  as  reliability  and 
provides  program  management  a  summary  of  the  project's 
progress  toward  a  specific  goal  for  that  factor  that 
was  set  at  the  outset  of  the  project. 

The  Metric  Report  provides  the  current  metric  scores. 
Developers  utilize  this  report  to  aid  in  design  and 
implementation  decisions. 

The  Exception  Report  is  utilized  by  Quality  Assurance 
personnel  to  identify  those  modules  which  vary  signifi¬ 
cantly  in  characteristics  from  the  average.  These 
modules  should  be  investigated  further  for  non- 
compliance  with  standards  and  conventions  or  potential 
problems. 

The  Summary  Report  is  utilized  by  test  personnel  to 
evaluate  the  amount  of  code  being  developed,  the  com¬ 
plexity  in  terms  of  number  of  paths,  the  number  of 
interfaces,  etc.,  any  of  which  have  an  influence  on 
test  strategy  and  effort. 

The  adherence  to  SQM  methodology  will  greatly  enhance  the 
effectiveness  of  the  tool  to  support  the  production  of 
quality  software  (Ref  88). 


Chapter  Summary 

The  software  quality  assurance  program  for  AFLC/hM 
should  utilize  tools  and  techniques  whose  cost/benefits  can 
be  demonstrated.  The  minimum  set  of  these  SQA  aids  that  is 
recommended  is  shown  in  Table  XVI.  The  SQA  organization 
should  have  a  toolsmith  whose  sole  responsibility  should  be 
the  development  and  maintenance  of  written  procedures  and  a 
tool  library.  Tools  placed  within  the  library  should  be 
qualified,  documented,  and  placed  under  configuration 
control.  Acquisition  of  new  tools  should  occur  only  when 
they  can  be  economically  justified. 
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Reifer  justified  the  inclusion  of  Software  Qiality 
Metrics  and  the  Automated  Measurement  Tool  in  a  SQA 
program.  He  stated,  "We  can  conclude  from  experience  that 
tools  and  techniques  improve  final  program  quality.  Before 
we  can  progress  further,  we  have  to  have  a  better  and  more 
quantitative  understanding  of  just  what  quality  is  and  how 
you  can  measure  it.  Only  then  can  we  hope  to  make  quality 
an  integral  part  of  the  software  systems  we 
produce."  (Ref  15)  SQM  and  the  AMT  provide  for  the 
quantitative  measurement  of  software  quality  that  is 
required. 

Chapter  V  provides  the  map  for  utilizing  SQM  and  the 
AMT  by  demonstrating  their  application  in  the  SDLC. 
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CHAPTER  V 


SDLC  DECISION-MAKING  MODEL:  A  MAPPING  TO  SQM 


It  is  an  accepted  proposition  that  man's  judgement  is 
no  better  than  his  information,  yet  the  magnitude  of  data 
generated  by  staffs  and  computers  usually  only  serves  to 
overwhelm  the  decision-making  process.  Hence,  there  is  a 

need  to  model  the  decisioning  process  within  the  software 

/ 

system  development  organization  to  determine  what 
information  is  needed  at  specific  points  during  the  SDLC. 

The  Software  Development  Life  Cycle 

Before  the  model  can  be  developed,  it  is  necessary  to 
further  describe  the  Software  Development  Life  Cycle  (SDLC) 
and  to  identify  the  deliverables  for  each  phase.  Gustafson 
and  Kerr  in  "Some  Practical  Experience  with  a  Software 
Quality  Assurance  Program"  in  the  Jan  '82  Communications  of 
the  ACM  noted  that  one  of  the  first  steps  in  establishing  a 
3QA  program  is  "to  define  the  development  life  cycle  used 
(or  preferred)  within  the  organization  sponsoring  a  new 
quality  assurance  program. ..,( and  then)  the  deliverables 
for  each  phase  must  be  identified  (Ref  56). 

Figure  7  provides  the  details  for  Software  System 
Development  Life  Cycle  (SDLC)  for  AFLC/LM.  Abstracting 
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Figure  7.  AFIC'S  Software  System  Development  life  Cycle 


information  from  the  June  1981  revision  of  AFLC/LM's 
Configuration  Management  chart  for  Automated  Data  Systems 
(ADS)  Development,  Figure  7  defines  the  phases  of  the  SDLC 
and  identifies  the  deliverable  products  for  each  phase. 
Moreover,  the  milestones  which  indicate  the  completion  of 
each  phase  are  also  identified.  For  example,  the  end  of 
the  Definition  Phase  is  marked  by  the  System  Design  Review 
(SDR),  which  is  a  milestone  of  the  SDLC.  The  primary 
deliverables  are  the  Functional  Description  and  the  Data 
Project  Plan.  Appendix  F  furnishes:  the  description  of 
each  phase  of  the  SDLC,  the  identification  and  contents  of 
the  deliverable  products,  and  the  significance  of  each 
milestone. 

The  deliverable  products  for  the  reviews  and  audits 
will  be  used  as  inputs  to  the  model  of  the  decision-making 
process  within  AFLC/LM's  software  development  organization. 

Modeling  the  Decisioning  Process 

A  basic  process  of  the  system  development  organization, 
which  is  a  basic  process  of  all  open  systems,  is  acquiring 
data  from  the  user  environment,  and  from  its  internal 
parts,  to  be  "consumed"  in  problem-definition  and 
decisioning  in  the  service  of  its  attempts  to  alter  its 
intended-states-of-af fairs,  its  internal  structure  or 
function,  for  some  aspect  or  domain  of  its 
environment  (Re)  53). 

This  process,  that  of  data  acquisition,  from  the 
environment,  constitutes  what  has  here  been  called  the 
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"scanning  process. "  The  system  must  be  related  to  the 
environment  through  two  kinds  of  channels:  the  afferent  (a 
scanning  system),  through  which  it  receives  information 
about  the  environment,  and  the  efferent  (decision  system), 
through  which  it  acts  on  the  environment  (Ref  54). 

The  relationships  between  the  software  system 
development  organization  (project  management)  and  user 
environment  are  shown  in  Figure  8.  For  logical 
completeness,  a  third  subsystem,  the  Intelligence  or 
Internal  Organizing  System,  has  been  added  to  reflect  an 
understanding  of  the  role  of  information  in  the 
decision-making  process.  Organizational  actions  which  are 
the  outputs  of  the  efferent  (decision)  subsystem  are  not 
based  upon  outputs  of  the  afferent  (scanning)  subsystem, 
but  rather  upon  the  outputs  of  the  intelligence  (internal 
organizing)  subsystem  which  acts  as  an  evaluator  of  the 
receptor  or  scanning  subsystem.  Raw  sensory  data  received 
by  the  scanning  (receptor)  system  is  eventually  fed  into 
the  decision  (efferent)  system  when  it  is  utilized  for 
problem-solving  purposes  (Re  49). 

The  next  step  of  this  sequence  is  a  selective 
conversion  and  organizat ion  of  the  data  >  into  a  form 
suitable  for  consumption  by  management.  It  is  only  through 
this  conversion  process  that  the  data  can  become 
information.  It  is  this  information  which  serves  as  a 
basis  for  decision-making  (Ref  49). 

It  must  be  emphasized  that  none  of  the  three  subsystems 
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(scanning,  organizing,  decision)  shown  in  Figure  8  operate 
independently  or  constitutes  a  separate  unit  within  the 
organization.  It  is  quite  likely  that  one  or  several 
individuals  within  the  system  development  organization  may 
be  engaged  in  any  or  all  three  of  the  activities  at  various 
times. 

This  interaction  system  has  been  treated  conceptually 
as  a  hierarchical  system.  A  hierarchical  system  is  one 


that  is 

composed 

of  interrelated 

subsystems. 

each 

subordinate 

to  the 

one  above  it 

(Ref  53). 

This 

"boxes-within-boxes "  way  of  looking  at  a  complex  phenomena 
i3  not  a  mere  partitioning  of  elements  but  one  joined  to 
the  relationships  of  the  several  parts  with  one  another  and 
with  the  whole.  This  interaction  is  regulated  by  the 
decision-making  process  (Ref  49). 

Moreover,  the  actions  of  the  software  system 
development  organization,  shown  in  Figure  8,  are  iterative 
in  nature  because  for  each  phase  of  the  SDLC  another 
iteration  (or  flow  through  the  three  subsystems)  occurs. 

In  other  words,  to  advance  from  one  phase  of  the  SDLC 
to  the  next,  a  decision  has  to  be  made.  This  decision 
determines  if  the  current  phase  is  complete  and  that  the 
development  effort  should  continue  to  the  next  phase,  or 
resolves  problems  in  that  current  phase. 

The  procedures  for  defining  software-oriented  quality 
attributes,  which  were  presented  in  Chapter  £1,  are  used 
within  this  model  to  determine  what  data  is  appropriate  and 
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Figure  8.  Software  System  Development  -  Environment  Interaction  System 
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to  convert  the  data  into  usable  information,  which  is,  in 
turn,  used  by  the  software  system  development  organization 
to  make  decisions  about  the  progress  of  the  development 
effort. 

Imposing  Metric  Framework  on  the  Model 

The  first  step  in  applying  the  framework  for  software 
quality  is  to  conceptualize  the  environment  in  which  the 
user's  system  will  be  operational.  It  is  important  to 
realize  that  individual  managers  within  the  user  group  will 
have  different  views  or  conceptualizations  of  the  user 
environment  (Ref  58).  Each  manager  has  a  different 
concept  of  what  the  constraints,  threats,  opportunities  and 
functions  of  the  user  environment  are.  Moreover,  this  view 
of  the  user  environment  is  constantly  changing  with  time 
because  of  changes  in  requirements,  regulations,  etc. 
Therefore,  it  is  important  to  establish  a  group  consensus 
of  the  user  environment  for  a  specific  period  or  interval 
of  time.  In  practice,  this  "interval  of  conceptualization" 
will  be  the  top  executive's  model  of  his  user  organization 
(hopefully  with  input  from  lower  level  managers). 

This  conceptualization  of  the  user  environment, 
establishes  the  desired  or  required  characteristics  of  the 
proposed  software  system  which  must  support  the  user 
operations. 


Table  IV  in  Chapter  II,  page  33,  provides  a  basic 
checklist  to  help  managers  identify  these  software  system 
characteristics.  Figure  9  illustrates  an  example  of 
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applying  this  software  metric  methodology. 

By  establishing  the  important  characteristics  of  the 
system  as  being  Real  Time  Application  and  having  a  Long 
Life  Cycle,  a  manager  (or  management  group)  can  then  use 
Table  III  in  Chapter  II,  page  32,  to  more  easily  isolate 
the  software  quality  factors  which  are  related  to  the 
system  characteristics. 

By  using  Tables  III  and  IV  in  interviews  with  AFLC/LM 
personnel,  six  software  quality  factors  initially  emerged 
as  candidate  choices.  From  the  system  characteristic  of  a 
Real  Time  Application,  three  factors  were  related: 
Reliability,  Efficiency,  and  Correctness.  Having  a  Long 
Life  Cycle  as  the  other  system  characteric  produced  three 
more  factors:  Maintainability,  Flexibility,  and 
Portability. 

Table  V  in  Chapter  II,  page  34,  was  used  to  tailor 
these  factors  to  AFLC's  cost  (and  environmental) 
constraints.  The  table  shows  that  the  factors  can  be 
grouped  according  to  system  life  cycle  activities 
associated  with  a  delivered  software  product.  The 
cost-to-implement  versus  life-cycle-cost-reduction 
relationship  exists  for  each  quality  factor.  It  was  this 
relationship  and  the  life  cycle  implications  that  made 
Reliability  and  Maintainability  the  two  critical  factors 
for  this  system,  G072A. 

Tables  VI  and  VII  in  Chapter  II  on  pages  35  and  36, 
further  substantiated  this  selection  of  Reliability  and 
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Maintainability  in  that  the  relationship  between  these  two 
factors  was  synergistic. 

Ratings  for  the  two  quality  factors  could  be 

established  using  Table  XIX.  For  example,  a  reliability 
rating  of  .99  requires  less  than  one  error  for  every  100 
lines  of  code  to  be  detected  during  formal  testing.  A 
maintainability  rating  of  . 8  requires  two  man-days  as  an 
average  level  of  maintenance  for  correcting  an  error. 
These  ratings  can  also  be  established  at  each  measurement 
period  during  the  development  as  follows: 

REQ  PDR  CDR  IMPL  ACCEPTANCE 
Reliability  .8  .8  .9  .9  .99 

Maintainability  .7  .7  .8  .8  .8 

The  progressively  better  scores  are  required  because  there 
is  more  detailed  information  in  the  later  phases  of  the 
development  to  which  to  apply  the  metrics  and  more 

confidence  in  the  metrics'  indication  of  quality.  This  is 
analogous  to  the  concept  of  reliability  growth  (Ref  28). 

The  next  step  in  establishing  the  software  quality 
requirements  specification  used  Table  VIII  in  chapter  II  on 
page  38  to  proceed  from  the  management-oriented  quality 
factors  to  the  software-oriented  criteria.  By  design,  the 
identification  of  these  criteria  was  automatic  and 
represented  a  more  detailed  specification  of  the  quality 
requirements . 

The  software  criteria  which  were  established  from 
Reliability  were:  Error  Tolerance,  Consistency,  Accuracy, 


Table  XIX 


and  Simplicity.  The  criteria  established  from 
Maintainability  were:  Consistency,  Simplicity. 
Conciseness,  Modularity,  and  Self-Descriptiveness.  The 
definitions  of  these  factors  and  criteria  are  provided  in 
Appendix  E.  These  definitions,  which  are  summarized  in 
Table  XX,  along  with  an  explanation  of  the  metrics  for  each 
of  the  criteria  (described  in  Appendix  C)  should  be 
provided  to  systems  analysts,  programmers  and  anyone  else 
involved  in  the  software  development  process. 

An  important  management  concept  was  realized  at  this 
initial  phase.  Gerald  Weinberg  demonstrated  that 
goal-directed  system  development  did  much  to  improve  the 
quality  of  the  software  system  (Ref  27).  By  stating 
up-front  the  specific  attributes  which  the  system  must 
exhibit,  project  management  can  greatly  influence  the 
effectiveness  of  the  end-product. 

Furthermore,  the  structured  methodology  of  the 
framework  for  Software  Quality  Metrics  enables  managers, 
who  know  little  about  software  but  much  about  what  is 
needed  to  support  the  user  group,  to  define  th<=ir 
requirements  specifications  in  software-oriented  terms.  At 
the  beginning  of  the  concepcual  phase,  software-oriented 
requirements  can  be  specified  in  the  Data  Automation 
Requirement  (DAR  )  and  the  Data  Project  Directive  (DPD). 

Monitoring  the  Development  effort 

The  framework  for  Software  Quality  Metrics  provides  the 
quality  assessment  capability  needed  by  management  in  its 
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decision-making  process.  Figure  10  superimposes  the 
application  of  Software  Quality  Metrics  onto  the 
decision-making  process.  It  represents  the  conceptual 
structure  of  the  decisioning  process  for  one  phase  of  the 
SDLC.  It  is  within  this  iterative  structure  that  the 
lowest  level  of  the  Software  Quality  Metrics  framework  — 
the  metrics  —  is  applied.  This  level  enables  the 
quantification  of  software  quality. 

Displayed  in  Figure  9,  the  software  quality  attributes, 
established  in  conceptualizing  the  system  requirements  of 
the  user  environment,  require  system  level  and  module  level 
collection  of  data  which  is  to  be  collected  during  the 
phases  of  the  SDLC.  The  specific  data  to  be  collected  is 
requested  on  metric  worksheets  listed  in  Appendix  A. 

A  Conceptual  Walk-Through 

Currently,  the  volumes  of  documentation  generated 
during  the  SDLC  under  Configuration  Management  serve  to 
overwhelm  project  management.  Indeed,  the  magnitude  of 
data  cannot  be  "consumed"  in  problem-definition  or  in 
decision-making. 


The  scanning 

( af  ferent ) 

system  must 

therefore 

be 

selective  in  its 

collection  of 

data,  and  should  have 

the 

capability  to 

ascertain 

what  data  is 

required 

for 

decisioning.  This  is  acquired  through  the  use  of  Software 
Quality  Metrics.  To  promote  an  understanding  of  how  the 
metrics  can  provide  the  assessment  of  software  quality,  a 
conceptual  walk-through  of  Figure  10  must  be  performed. 
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Figure  10.  SQM  Applied  to  the  Decision-Making  Process 


Input  to  the  scanning  system  is  the  raw  sensory  data 
generated  by  software  development  documentation,  source 


code,  etc.  The  metric  worksheets  provide  the  selection 
process  by  specifying  what  data  to  obtain.  The  up-front 
specification  of  software  factors  and  attributes  has 
narrowed  the  scope  of  data  to  be  collected.  Specifically, 
by  determining  Reliability  and  Maintainability  as  the 
software  factors,  metric  data  is  collected  which  assesses 
the  software  attributes:  Consistency,  Simplicity, 
Conciseness,  Modularity,  Self-Descr.  iptiveness,  Error 
Tolerance,  and  Accuracy.  Though  other  metrics  should  also 
be  collected,  management  attention  can  be  focusing  on  the 
achievement  of  Reliability  and  Maintainability. 

Moreover,  the  metric  worksheets  serve  as  feedback  to 
insure  that  documentation  is  complete,  i.e.,  confirming 
that  the  information  that  is  requested  by  the  worksheets  is 
actually  found  in  the  system  documentation.  This  should 
warn  project  management  that  documentation  is  incomplete, 
thus  allowing  early  detection  of  a  problem.  Also  serving 
as  a  checklist  to  insure  that  all  metric  data  is  collected, 
the  worksheets  function  as  the  output  of  the  scanning 
system  and  as  the  input  to  the  organizing  (intelligence) 
system. 

The  process  internal  to  the  organizing  system  is  the 
actual  update  of  the  Automated  Measurement  Tool  (AMT)  data 
base.  In  turn  the  AMT  automatically  computes  the  metri<~ 
scores  by  using  the  algorithms  listed  in  Appendix  D.  The 
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AMT  provides  a  report  generation  function,  which  was 
discussed  in  Chapter  IV,  that  can  be  used  to  provide 
specific  information  to  project  management,  the  system 
analysts  and  programmers,  the  SQA  staff,  the  test  group, 
and  researchers.  The  nine  reports  it  generates  (Worksheet, 
Summary,  Matrix,  Statistics,  Quality  Growth,  Normalization, 
Module,  and  Exception  Reports)  can  be  complemented  by 
organizational  forms  specific  for  AFLC/LM. 

These  forms  provide  the  basis  for  feedback  in  resolving 
inconsistencies  in  actual  metric  scores  to  what  may  have 
been  expected.  This  provides  direction  to  the  development 
staff  to  look  in  specific  areas  of  the  programs  for 
possible  problems.  As  output  of  the  organizing  system, 
these  forms  provide  the  input  to  the  decision  (efferent) 
system. 

At  this  point,  management  is  in  a  better-informed  state 
to  make  decisions  about  the  quality  of  the  system.  The 
forms  and  explanations  provide  the  decision  system  with 
information  (not  raw  data)  about  possible  cost  trade-offs, 
various  development  alternatives,  and  the  quality  of  the 
software.  The  feedback  would  be  to  determine  the  specific 
information  that  could  further  enhance  the  decision-making 
process. 

The  output  is  the  decision:  to  contii ue  on  to  the  next 
phase,  to  repeat  portions  of  the  current  phase  or  to 
fall-back  onto  a  portion  of  a  previous  phase  to  change  a 
part  of  the  development  effort.  For  example,  ii  an 
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opportunity  could  be  realized  by  changing  a  requirement, 
even  though  the  development  effort  is  in  the  design  phase, 
then  some  of  the  procedures  would  have  to  be  duplicated  and 
a  pass  through  the  three  systems  (scanning,  organizing,  and 
decision)  would  insure  completeness  of  the  change. 

It  is  anticipated  that  by  accomplishing  the 
structured-flow  through  the  scanning,  organizing  and 
decision  systems  and  by  satisfying  the  information 
requirements  of  the  Software  Quality  Metrics,  that  the 
appropriate  decision  at  the  end  of  each  phase  will  be  to 
continue  to  the  next  phase. 

Metric  Application  Throughout  the  SDLC 

Each  phase  of  the  System  Development  Life  Cycle  is  an 
iteration  of  Figure  10;  however,  there  is  a  sufficient 
amount  of  variation  among  the  phases  to  warrant  a 
discussion  of  the  flow  through  the  entire  SDLC.  The  entire 
structural  model  of  the  SDLC  superimposed  by  the  framework 
for  Software  Quality  Metrics  is  displayed  at  the  end  of 

a 

this  chapter  in  Figure  15. 

The  procedures  for  describing  the  software-oriented 
attributes  of  the  system,  which  are  depicted  in  Figure  9, 
remain  the  same  for  each  system  development. 

Requirements  Analysis  Phase.  Represented  in  Figure 


11,  the  Requirements  Analysis  is  the  first  phase  of  the 
SDLC  and  is  called  the  Conceptual  Phase  under  Configuration 
Management  at  AFLC/LM.  At  the  end  of  this  phase,  the 
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Figure  11.  Requirements  Anali 
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dec is ion -making  process  is  extremely  important  because  the 
Software  Requirements  Review  ( SRR  ) ,  whi<  h  establish ?s  the 
Functional  Baseline,  serves  as  the  milestone  that  also 
represents  the  beginning  of  the  )efinition  Phase. 
Therefore,  the  data  which  serves  as  the  basis  for  the 
decision-making  process  plays  a  key  role  in  determining  the 
completeness  of  the  Requirements  Analysis. 

The  primary  input  to  the  scanning  system  during  the 
Requirements  Analysis  is  the  documentat  on  specified  for 
the  SRR  under  Configuration  Management.  Other  documents, 
such  as  the  Data  Automation  Requirements  ( DRR )  and  Data 
Project  Directive (DPD )  also  serve  as  sources  for  input 
data.  The  scanning  process  is  the  manual  collection  of 
system  level  metrics  as  specified  by  Metric  Worksheet  #1  in 
Appendix  A. 

Because  it  was  decided  up-front  that  the  software 
system  must  exhibit  the  quality  factors  of  Reliability  and 
Maintainability,  the  data  collected,  which  should  be 
available  during  the  systems  requirements  analysis,  is 
information  which  checks: 

if  error  analysis  has  been  performed  and  budgeted  to 
functions ; 

if  there  are  definitive  statements  of  the  accuracy 

requirements  for  inputs,  output,  processinq  and 
constants ; 

if  there  are  definitive  statements  of  the  error  toler¬ 
ance  of  input  data; 

if  there  are  definitive  statements  of  the  requirements 
for  recover  -  from  computational  failures; 

if  there  is  a  definitive  statement  of  the  requirements 
for  recovery  from  hardware  faults; 

if  there  is  a  definitive  statement  of  the  reqj irements 
for  recover/  from  device  errors; 

the  number  of  major  functions  and  data  references;  etc. 
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The  Software  Quality  Metric  Worksheet  #1  provides  the 
feedback  to  project  management  and  the  SQA  function  by 
providing  a  checklist  to  insure  that  a  complete  analysis  of 
requirements  has  been  conceptualized  and  documented.  The 
Metric  Worksheet  #1  functions  as  the  output  document  of  the 
scanning  system  and  as  the  input  document  of  the  organizing 
system  within  the  Requirements  Analysis  Phase  of  the  SDLC. 

The  process  within  the  organizing  (intelligence)  system 
updates  the  Automated  Measurement  Tool  (AMT)  data  base  and 
investigates  inconsistencies  within  documentation, 
requirements  analysis,  and  worksheet  specifications.  For 
example,  certain  requirements  of  the  system  may  be  in 
conflict  with  the  attributes  which  seek  to  insure  that  the 
system  will  be  Reliable  and  Maintainable.  These  findings 
can  be  discussed  on  an  organizational  form  which  explains 
these  possible  inconsistencies. 

This  organizational  form  serves  as  the  primary  output 
of  the  organizing  system  and  the  input  for  the  decision 
system.  Although  the  metrics  reports  can  be  generated  by 
the  AMT,  they  provide  very  little  useful  information  this 
early  in  the  SDLC  (during  requirements  analysis).  The 
process  involved  within  the  decision  system  is  to  determine 
the  significance  of  the  findings  generated  by  the 
organizing  system.  Cost  trade-offs  should  be  considered  in 
keeping  the  requirements  as  they  are,  or  to  modify  the 
system  requirements  to  insure  high  Reliability  and 
Maintainability.  It  is  the  decision  system  which  provides 


* 


95 


r 


the  feedback  to  determine  the  adequacy  of  the  system 

requirements  and  documentat ion. 

By  utilizing  the  feedback  within  and  between  the 
subsystems  and  applying  the  metric  concepts,  it  is 
anticipated  that  the  logical  output  of  the  decision  system 
will  be  to  continue  to  the  Design  Phase  of  the  SDLC. 

Design  Phase.  Represented  in  Figure  12,  the  Design 

Phase  is  the  second  phase  of  the  SDLC,  and  it  is  the 

equivalent  of  the  Definition  through  Detail  Design  Phase 
under  Configuration  Management  for  AFLC/LM.  This  phase 
plays  an  important  role  in  the  SDLC  decisioning  process 
because  the  System  Design  Review  (SDR)  which  establishes 
the  Allocated  Baseline,  the  Preliminary  Design  Review 

(PDR),  and  the  Critical  Design  Review  (CDR )  all  serve  as 
decision  milestones.  Therefore,  the  data,  that  is  made 
available  on  Design  Phase  documentation  and  serves  as  the 
basis  for  the  decision-making  process,  plays  a  key  role  in 
determining  the  completeness  of  the  Design  Phase. 

Each  review  (SDR,  PDR,  and  CDR)  represents  another 
iteration  of  the  flow  through  this  phase.  In  the  true 
sense,  this  is  a  recursive  phase  --  it  repeats  itself  in 
the  collection  of  data  during  each  design  review. 
Therefore,  the  input  to  the  scanning  system  during  the 
Design  Phase  depends  upon  the  particular  review  or 
iteration  of  this  phase. 

The  scanning  process  is  the  manual  collection  of  both 
system  level  and  module  level  metrics  as  specif  Led  by 
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Metric  Worksheets  #2a  and  #2b  which  are  listed 


in 


Appendix  A.  It  should  be  noted  that  if  a  standard  design 
language  such  as  PDL  (Program  Design  Language)  is  used, 
then  the  AMT  can  be  used  to  collect  the  data.  The 
efficiency  of  this  automatic  collection  of  data  not  only 
saves  time  and  human  resources  but  comes  close  co  providing 
real-time  feedback  on  the  computation  of  metric  scores. 

Because  it  was  decided  up-front  that  the  software 
system  should  be  highly  reliable  and  maintainable,  specific 
data,  which  should  be  available  during  the  design  phase,  is 
collected  at  both  the  system  level  and  the  module  level,  as 
indicated  by  the  metric  worksheets. 

The  data  which  the  Metric  Worksheet  #2a  collects  is 
system  level  information  which  checks: 


if  match  library  routines  to  be  used  have  been  checked 
for  sufficiency  with  regards  to  accuracy  require¬ 
ments  ; 

if  concurrent  processing  is  centrally  controlled; 
if  error  conditions  are  reported  by  the  system; 
if  errors  are  automatically  fixed  or  bypassed  and  if 
processing  continues; 
if  errors  require  operator  intervention; 
if  provisions  for  recovery  from  hardware  faults  and 
device  errors  are  provided; 
if  a  hierarchy  of  system,  identifying  all  modules  in 
the  system,  is  provided; 

if  the  number  of  modules  is  recorded;  if  so,  what  is 
the  number  of  modules. 


The  data  which  Metric  Worksheet  #2b  collects  is  module 
level  information  (collected  from  each  module)  which  checks 
if: 


numerical  techniques  being  used  in  algorithms  have 

been  analyzed  with  regard  to  accuracy  requirements; 
values  of  inputs  range  have  been  tested; 
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conflicted  requests  and  illegal  combinations  have  been 
identified  and  checked; 

there  is  a  check  to  see  if  all  necessary  data  is  avail¬ 
able  before  processing  begins; 

all  input  is  checked,  reporting  all  errors  before  pro¬ 
cessing  begins; 

loop  and  multiple  transfer  index  parameters  range  are 
tested  before  use; 

subscripts  range  are  tested  before  use; 

outputs  are  checked  for  reasonableness  before  pro¬ 
cessing  continues; 

etc. 


The  Design  Phase  metric  worksheets  provide  feedback  to 
project  management  and  the  SQA  function  by  providing  a 
checklist  to  insure  that  the  design  specif ications 
adequately  describe  how  the  system  requirements  will  be 
achieved.  Moreover,  the  worksheets  insure  that  the 
necessary  design  information  is  documented.  Worksheets  #2a 
and  #2b  function  as  the  output  documentation  of  the 
scanning  system  and  as  the  input  documentation  of  the 
organizing  system  within  the  Design  Phase  of  the  SDLC. 

The  process  within  the  organizing  (intelligence)  system 
updates  the  AMT  data  base  and  derives  metric  scores.  It 
also  seeks  to  explain  metric  anamolies  --  inconsistenc ies 
with  historical  data  versus  current  metric  scores.  The 
reports  that  are  generated  by  the  AMT  provide  useful 
information  even  in  this  early  phase.  Organizational  forms 
should  be  generated  to  document  any  inconsistencies  and  to 
provide  explanations  of  the  metric  reports.  These  forms 
will  be  used  with  the  metric  reports  as  output  of  fhe 
organizing  system  and  as  input  to  the  decision  system.  The 
decision  system  determines  the  significance  of  the  metric 
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reports  and  organizational  reports.  Again  trade-offs  must 
be  considered,  and  a  determination  of  the  adequacy  of 
design  specifications  and  documentation  can  be  challenged. 
By  using  the  feedback  within  and  between  the  subsystem  and 
applying  the  metric  concepts,  it  is  anticipated  that  the 
logical  output  of  the  decision  system  will  be  to  continue 
to  the  Implementation  (coding  &  checkout)  Phase. 

Inplementation  Phase.  Represented  in  Figure  13,  the 
Coding  and  Checkout  Phase  is  the  third  phase  in  the  SDLC. 
This  phase  is  the  equivalent  of  the  Development  (Code 
through  Subsystem  Test)  Phase  under  Configuration 
Management  for  AFLC/LM. 

Detailed  Design  Specifications  from  documentation  which 
resulted  from  the  CDR  serve  as  the  preliminary  data  input 
to  the  scanning  system.  Once  programming  has  begun,  the 
source  code  provides  data  input.  During  the  subsystem  or 
unit  test,  the  Preliminary  Functional/Physical 
Configuration  Audits  ( FCA/PCA )  and  Product  Verification 
Review  (PVR)  represent  key  decision-making  points  during 
the  SDLC.  As  a  result  of  the  PVR,  the  Product  Baseline  is 
established.  Therefore,  the  input  to  the  scanning  system 
during  the  Implementation  Phase  depends  upon  the  particular 
iteration  (review,  audit  or  code)  of  this  phase. 

The  scanning  process  is  the  manual  and  AMT  collect  of 
module  level  metrics  as  specified  by  Metric  Worksheets  #3 
and  #2b  which  are  listed  in  Appendix  A.  If  a  standard 
design  language  is  used,  the  AMT  can  be  used  to 
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automatically  collect  the  detailed  design  data.  The  AMT 

automatically  collects  source  code  data  for  module  level 

metrics.  A  metric  worksheet  is  used  for  every  module  to 

insure  all  metric  data  is  collected.  As  a  module  is 

changed.  Worksheet  #2b  must  also  be  updated,  thus  replacing 

the  previous  Worksheet  #2b  for  that  module. 

To  insure  that  software  exhibits  Reliability  and 

Maintainability,  specific  data  must  be  present.  The  metric 

data  which  Worksheet  #3  collects  from  each  module  checks: 

the  number  of  decision  points; 

the  number  of  declarative  statements; 

the  number  of  lines  excluding  comments; 

the  number  of  statement  labels; 

the  number  of  conditional  branches; 

the  number  of  unconditional  branches; 

the  number  of  loops ; 

the  number  of  lines  of  comments; 

if  there  are  prologue  comments  containing  information 
about  the  function,  author,  version  number,  date, 
inputs,  outputs,  assumptions  and  limitations; 
the  number  of  decision  points  and  transfers  of  control 
that  are  not  commented; 
if  all  machine  language  code  is  commented; 
etc. 

The  data  on  Metric  Worksheet  #2b  needs  to  be  updated  as 

the  modules  are  changed  checks  if: 

numerical  techniques  being  used  in  algorithms  have 

been  analyzed  with  regard  to  accuracy  requirements; 
values  of  inputs  range  have  been  tested ; 
conflicted  requests  and  illegal  combinations  have  been 
identified  and  checked; 

there  is  a  check  to  see  if  all  necessary  data  is  avail¬ 
able  before  processing  begins; 
all  input  is  checked,  reporting  all  errors  before  pro¬ 
cessing  begins; 

loop  and  multiple  transfer  index  parameters  range  ar^ 
tested  before  use; 

subscripts  range  are  tested  before  use; 
outputs  are  checked  for  reasonableness  before  pro¬ 
cessing  continues; 

etc. 
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The  X  implementation  Phase  metric  worksheets  provide 
feedback  to  project  management  and  the  SQA  Function  by 
providing  a  checklist  to  insure  that  the  coding  and 
checkout  procedures  adequately  produce  the  specified 
software  system.  Moreover,  the  worksheets  insure  that  the 
necessary  development  information  is  documented. 
Worksheets  #2a  and  #3  for  each  module  function  as  the 
output  documentation  of  the  scanning  system  and  as  the 
input  documentation  to  the  organizing  system  within  the 
Implementation  Phase  of  the  SDLC. 

The  process  within  the  organizing  ( intelligence )  system 
updates  the  AMT  data  base,  derives  metric  scores,  and  seeks 
to  explain  metric  anamolies.  The  reports  that  are 
generated  by  the  AMT  provide  a  quantified  assessment  of 
software  quality.  Organizational  forms  should  be  generated 
to  document  any  inconsistencies  and  to  provide  explanations 
for  the  metric  scores.  These  forms  will  then  be  used  with 
the  metric  reports  as  the  output  of  the  organizing  system 
and  as  the  input  to  the  decision  system  of  the 
Implementation  Phase. 

The  decision  system  determines  the  significance  of  the 
metric  reports  and  organizational  forms.  At  this  point  an 
adequacy  of  development  documentation  can  be  challenged  -- 
the  metric  reports  will  show  the  problem  areas.  Management 
can  inquire  of  the  amount  of  code  coverage  (number  of  paths 
executed)  and  determine  if  all  code  can  be  reached.  3y 
using  the  feedback  within  and  between  the  subsystems  and 
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applying  the  metric  concepts,  it  is  anticipated  that  the 
logical  output  of  the  decision  system  will  be  a  smooth 
transition  into  the  Test-and-Integration  Phase. 

Test  and  Integration  Phase.  Represented  in  Figure  14, 
the  Test-and-Integration  is  the  fourth  phase  in  SDLC.  This 
phase  is  the  equivalent  of  the  test  phase  (and  System  Test 
in  the  Development  Plan)  under  Configuration  Management  for 
AFLC/LM. 

The  subsystem/system  test  results  and  PVR  serve  as  the 
preliminary  input  to  the  scanning  system.  The 
documentation  associated  with  the  final  FCA/PCA  is  also 
available  during  the  initial  part  of  the  test  phase. 
Finally  the  System  Validation  Review  (SVR),  which 
represents  a  key  decision-making  point  of  the  SDLC,  marks 
che  end  of  the  test  phase.  Therefore,  the  documer  tation 
associated  with  the  SVR  must  detail  all  the  data  which  is 
necessary  to  insure  a  reliable  and  maintainable  software 
system.  Because  this  becomes  available  at  different 
pointsduring  the  test  phase,  the  input  to  the  scanning 
system  during  the  test  phase  depends  upon  the  particular 
iteration  (flow  through)  of  this  phase. 

The  scanning  process  can  rely  mainly  on  the  AMT  to 
collect  the  module  level  metrics  as  specified  by  Metric 
Worksheet  #3,  which  is  updated  from  the  implementation 
phase.  Because  this  phase  is  testing  for  the  total  system 
integration,  it  is  necessary  to  update  Metric  Worksheet  #2a 
to  insure  that  any  changes  to  modules  during  the 
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Implementation  Phase  did  not  adversely  affect  the  system 
level  performance.  The  updates  to  the  metric  worksheets 
provide  the  checklist  to  insure  necessary  documentation  and 
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organizing  system  within  the  Test-and-Integration  phase  of 
the  SOLC. 

The  process  within  the  organizing  (intelligence)  system 
updates  the  AMT  data  base,  derives  metric  scores,  and 
determines  potential  problem  areas  (modules).  The  reports 
that  are  generated  by  the  AMT  provide  a  quantified 
assessment  of  the  software  system  quality.  Organizational 
forms  should  document  the  potential  problems  within  modules 
identified  by  the  metric  reports.  These  forms  will  then  be 
used  with  the  metric  reports  as  the  output  of  the 
organizing  system  and  as  the  input  to  the  decision  system 
of  this  phase. 

The  decision  system  determines  it  the  system  satisfies 
the  specified  levels  of  quality,  as  stated  during  the 
requirements  analysis.  The  decision-maker s  must  decide  if 
the  potential  problem  areas  should  be  investigated  or  to 
simply  identify  the  modules  and  allow  the  system  to  go  into 
operation.  By  using  the  feedback  within  and  between  the 
subsystems  and  applying  the  metric  concepts,  it  is 
anticipated  that  the  logical  output  of  the  decision  system 
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will  be  to  allow  the  system  to  become  operational. 

Figure  15  shows  the  interrelationship  between  the 
phases  by  indicating  the  possible  flows  through  the  SDLC. 
Because  each  phase  has  already  been  depicted  in  Figures  11 
through  14,  the  legibility  of  each  element  is  not 
important.  The  significance  of  Figure  15  is  to  demonstrate 
that  the  entire  SDLC  is  iterative:  processes  are  repeated 
within  phases,  phases  can  be  repeated,  and  decisions  can  be 
made  to  "fall  back"  to  any  previous  phase.  It  is  necessary 
to  understand  that  changes  introduced  later  in  the  life 
cycle  require  a  flow  through  the  entire  model.  Only  then 
can  the  developer  insure  a  "complete"  software  system. 

Chapter  Summary 

Conceptualized  within  the  context  of  the 
decision-making  process,  software  development  has  been 
portrayed  as  an  iteration  of  subsystems  within  the  various 
phases  of  the  SDLC  that  require  specific  information  to 
enhance  decisioning.  The  utilization  of  the  SQM  concepts 
provides  a  quantitative  assessment  of  software  quality  for 
a  goal-directed  system  development  which  is  controlled  by 
management. 

Chapter  VI  develops  the  checklists  which  enhance  the 
SQM  concepts  by  providing  prescriptive  elements  which  are 
indicators  of  reliable  and  maintainable  software. 
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CHAPTER  VI 


SQA  INFORMATION  REQUIREMENTS:  A  CHECKLIST 


To  insure  a  quality  software  system,  the  deliverable 
products  must  provide  information  to  demonstrate  that  thi 
software  system  exhibits  the  desired  quality  attributes. 
The  metrics,  which  establish  the  quantitative  assessment  of 
that  software  quality,  are  not  based  on  the  availability  of 
specific  documentation,  but  rather  on  the  availability  of 
key  information.  With  this  in  mind,  it  is  the  intent  of 
this  chapter  to  satisfy  the  third  goal  of  this  thesis:  to 
demonstrate  how  the  SQM  concepts  can  be  applied  at  AFLC. 
This  is  accomplished  by  providing  checklists  for  system 
developers.  The  checklists  help  to  insure  that  key 
information  is  made  available  during  the  SOLC.  Because  the 
checklists  use  items  from  the  SQM  worksheets,  which  are 
descriptive  in  nature,  they  expand  the  SQM  concepts  by 
providing  a  more  prescriptive  means  for  developing  quality 
software. 

Documentation  Requirements 

One  of  the  main  considerations  in  applying  Software 
Quality  Metrics  is  the  availability  of  data  or  software 
products  which  provide  the  sources  for  collection  of  the 
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metrics.  Software  products  include  the  source  code  (the 
most  obvious  and  researched  source  of  metrics  to  date), 
documentation  including  requirements  specifications,  design 
specifications,  manuals,  test  plans,  problem  reports, 
correction  reports,  and  reviews  (Ref  24). 

There  are  basic  documentation  requirements  regardless 
of  the  size  or  cost  of  a  program.  Certainly,  less 
documentation  is  produced  during  a  low  cost/short  life 
project,  but  it  should  still  contain  certain  information. 
The  documents  may  not  be  in  as  much  detail,  yet  they  still 
must  contribute  to  the  quality  of  the  software  product. 
Indeed,  this  is  the  situation  for  AFLC.  The  program 
classification  dictates  the  documentation  requirements. 
Major  programs  are  better  documented  because  of  upper 
management  attention.  For  example,  MMSIP  (Maintenance 
Management  Systems  Inprovement  Project)  and  its  subsystems, 
received  upper  management's  attention  because  they  provide 
cost  information  about  organic  depot  maintenance  to  aid  in 
managing  the  Depot  Maintenance  Services  Branch  of  the  Air 
Force  Industrial  Fund  (DMS,  AFIF).  Because  of  this 
attention,  G072A  MMSIP,  which  is  a  subsystem  of  MMSIP,  is 
supposed  to  be  one  of  the  better  documented  systems  at 
AFLC  (Ref  51). 

However,  it  was  difficult  to  extract  various 
data/information  because  the  documentation  either  did  not 
provide  the  information,  or  did  not  clearly  identify  the 
data.  Now  this  point  is  critical  in  considering 
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establishing  Software  Quality  Metrics  and  the  Automated 
Measurement  Tool.  Though  the  format  was  in  compliance  with 
regulations,  the  availability  of  key  information  wes  not 
made  readily  apparent. 

To  insure  a  quality  product,  the  documentation  cf  that 
software  must  provide  key  information  related  to  the 
desired  quality  attributes.  It  is  an  organizational 
decision  to  determine  the  placement  of  the  informaticn  into 
specific  documents. 

To  aid  developers  in  documenting  a  software  system's 
reliability  and  maintainability,  a  checklist  of  standards 
has  been  developed  for  each  phase  of  the  SDLC.  These 
checklists  provide  goal-directed  documentation  procedures 
because  developers  are  aware  of  the  specific  information 
requirements,  not  just  format. 

Application  for  Checklists  with  SQM 

If  the  development  team  were  to  use  the  checklists  in 
this  chapter  with  existing  Question  Sets  listed  in  AFLC's 
Project  Manager's  Handbook  (Working  Copy),  then  it  would  be 
combining  software-oriented  requirements  of  the  checklists 
with  the  user  or  management-oriented  requirement;,  of  the 
Question  Sets.  To  demonstrate  the  uses  of  the  checklists 
and  SQM  for  the  G07  2A  MMStP,  one  must  consider  what  would 
have  happened.  This  is  easiest  to  visual iz»  by  applying 
the  model  developed  in  Chapter  V  to  illustrate  deficiencies 
in  current  development  practices.  Because  the  AMT  is  not 
yet  available  to  AFLC,  it  was  infeasible  to  collect  module 
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level  information. 


Therefore,  it  is  only  possible  to 


illustrate  the  first  two  phases  of  the  SDLC. 

Conceptual  Phase.  At  the  beginning  of  the  Conceptual 
Phase  several  documents  were  generated  to  reflect  the 
requirements  of  the  user.  The  Environmental 
Assessment/Statement  accompanied  the  Data  Automation 
Requirement  (DAR)  through  the  decision-making  process  to 
HQ  USAF.  It  was  during  this  initial  phase  that  the 
end-user  identified  the  system  characteristics.  It  was 
known  that  the  system  would  have  a  long  life  cycle  and  that 
real  time  application  was  needed.  Because  maintainability 
had  become  a  command  level  requirement  (Ref  89),  it  was 
easy  to  identify  it  as  a  required  software  factor. 
However,  there  was  r.o  further  definition  of  software 
attributes. 

By  using  the  step-wise  development  from  system 
characteristics  to  software  attributes,  depicted  earlier  in 
Figure  9  on  page  82,  AFLC' s  project  management  could  have 
further  definitized  system  requirements.  Moreover,  it 
could  have  specified  a  Reliability  rating  and  a 
Maintainability  rating  as  shown  in  Table  XIX  on  page  85. 
This  could  have  provided  up-front  criteria  on  which  to  base 
final  acceptance  of  the  system  during  the  testing  phase. 

This  failure  to  provide  software-oriented  requirements 
meant  that,  at  the  time  of  the  Systems  Requirements 
Review  (SRR),  the  system  developers  were  unable  to  assess 
the  system's  reliability  and  maintainability.  If  the  .sqm 
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worksheets  would  have  been  applied  at  that  time,  only  the 
items  which  corresponded  to  Computer  Program  Configuration 
Items  (CPCI )  would  have  been  satisfied.  For  example,  the 
description  of  the  flow  of  processing,  and  all  decision 
points  in  that  flow,  was  provided  under  CM,  however,  items 
relating  to  error  tolerance  were  not  even  addressed;  so 
there  was  a  gap  in  current  procedures  under  CM. 

This  gathering  of  data  would  complete  the  scanning 
phase  (system)  of  the  Requirements  Analysis  shown  in 
Figure  11  on  page  93.  The  immediate  need  for  a  checklist 
can  now  be  seen  as  a  way  to  resolve  the  inconsistencies 
characterizing  current  documentation  requirements  under  CM 
to  that  of  a  more  desired  development  effort. 

Therefore,  to  enhance  the  Question  Sets  in  the  Project 
Manager's  Handbook  and  the  procedures  under  CM,  a  checklist 
has  been  developed  that  will  make  system  designers  aware  of 
items  that  are  present  in  reliable  and  maintainable  systems 
during  the  Conceptual  Phase.  At  the  beginning  of  the 
Requirements  Analysis,  system  designers  should  be  given 
this  checklist  to  insure  that: 

1.  Requirements  are  itemized  so  that  the  various 

functions  to  be  performed  (their  input  and 
outputs)  are  clearly  delineated; 

2.  All  data  references  are  defined; 

3.  All  defined  functions  are  justified; 

4.  All  referenced  functions  are  defined; 

5.  All  data  referenced  are  used; 

6.  A  description  is  provided  for  the  i low  of 

processing  and  all  decision  points  in 
that  flow; 
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7.  All  problem  reports,  related  to  the  requirements 

are  recorded  and  closed  (resolved)  prior 
to  the  System  Requirements  Review  (SRR); 

8.  An  error  analysis  has  been  performed  and  budgeted 

to  functions; 

9.  There  are  definitive  statements  about  the  accuracy 

requirements  for  inputs,  outputs,  processing, 
and  constants; 

10.  There  are  definitive  statements  of  the  error 

tolerance  of  input  data; 

11.  There  are  definitive  statements  of  the  requirements 

for  recovery  from:  (1)  computational 
failures,  (2)  hardware  faults,  and  (3)  device 
errors. 

Only  after  this  additional  data  has  been  gathered  can  the 
organizing  system  provide  adequate  information  to  the 
decision  system.  Without  this  additional  data,  a  decision 
should  not  have  been  made  to  continue  on  to  the  next  phase 
of  development,  for  the  Requirements  Analysis  was  truly 
incomplete. 

Design  Phase.  However,  the  development  effort  did 
continue  into  the  Design  (Definition  through  Detail  Design) 
Phase.  Again,  the  only  data  that  could  be  gathered  on  the 
worksheets  by  the  scanning  system  was  that  which 
corresponded  to  CPCI's. 

The  following  checklist  has  been  developed  to  fill  this 
gap  of  information  requirements.  This  information  should 
appear  in  the  draft  of  the  user's  manual.  °rior  to 
beginning  the  Design  Phase,  system  developers  should  insure 
that : 
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1.  There  is  a  matrix  relating  itemized  requirements 

to  modules  which  implement  those 
requirements ; 

2.  All  major  functions  are  identified,  defined,  and 

used ; 

3.  All  interfaces  between  functions  are  defined; 

4.  All  problem  reports  are  resolved; 

5.  A  profile  of  the  number  of  problem  reports  is 

"broken-out"  by  the  following  types — 
Computational,  Logic,  I/O,  Data  Handling, 
OS/System  Support,  Configuration, 
Routine/Routine  Interface,  Tape  Processing, 
User  interface.  Data  base  interface.  User 
requested  changes,  Preset  data.  Global 
variable  definition.  Recurrent  errors, 
Documentation,  Requirement  compliance. 
Operator,  Questions,  Hardware; 

6.  Math  library  routines  to  be  used  have  been  checked 

for  sufficiency  with  regard  to  accuracy 
requirements ; 

7. —  .Concurrent  processing  is  centrally  controlled; 

8.  All  error  conditions  a.e  reported  by  the  system, 

and  determine  how  many  of  these  errors  are 
automatically  fixed  or  bypassed  and  allow 
processing  to  continue,  and  how  many  require 
operator  intervention; 

9.  Provisions  for  recovery  from  hardware  faults  and 

device  errors  is  provided; 

10.  A  system  hierarchy  is  provided,  identifying  all 

modules  in  the  system; 

11.  All  modules  and  duplicate  functions  are 

identified  ; 

12.  All  modules  called  by  more  than  one  other  module 

are  identified; 

13.  The  constants  used  in  the  system  are  defined  only 

once. 

For  each  module,  system  developers  must  check  to  insure 
that  the  following  guidelines  are  contained  within  the 
system/ subsystem/prog  ram  specifications  and  test  plan 


documentation : 


1.  Inputs,  outputes,  and  functions  being  performed 

can  be  clearly  identified; 

2.  A  minimal  number  of  data  references  are  defined, 

computed  or  obtained  from  external  sources; 

3.  All  conditions  and  processing  are  defined  for  each 

decision  point; 

4.  The  type  of  problem  report  is  being  identified 

within  a  profile  of  problem  reports; 

5.  When  an  error  condition  is  deteted,  a  message  is 

passed  to  the  calling  module; 

6.  Numerical  techniques  used  in  algorithms  are 

analyzed  with  regards  to  accuracy  require¬ 
ments  ; 

7.  values  of  input  ranges  are  tested; 

8.  Conflicting  requests  and  illegal  combinations 

are  identified  and  checked; 

9.  There  is  a  check  to  see  if  all  necessary  data  is 

available  before  processing  begins; 

10.  All  input  is  checked,  reporting  all  errors  before- 

processing  begins; 

11.  Loop  and  multiple  transfer  index  parameters  range 

are  tested  before  use; 

12.  Subscripts  range  are  tested  before  use; 

13.  Outputs  are  checked  for  reasonableness  before 

processing  continues; 

14.  The  number  of  decision/subdecision  points  is  known; 

15.  The  number  of  conditional  and  unconditional 

branches  is  identified; 

16.  The  module  is  not  dependent  on  information  of 

prior  processing; 

17.  Any  limitations  of  the  processing  that  are 

performed  by  the  module  are  identified; 

18.  The  number  of  entrances  into  modules  and  number 

of  exits  from  modules  are  known; 


116 


19.  The  number  of  references  to  system  library 

routines,  utilities  or  other  system 

provided  facilities  is  known; 

20.  The  number  of  I/O  actions  and  the  number  of 

calling  sequence  parameters,  which  are 

control  variable,  is  known; 

21.  Input  is  passed  as  calling  sequence  parameters; 

22.  Output  is  passed  back  to  calling  module; 

23.  Control  is  returned  to  calling  module; 

24.  Temporary  storage  is  not  shared  with  other 

modules ; 

25.  A  module  does  not  mix  input,  output,  and 

processing  functions  in  the  same  module; 

26.  The  number  of  machine  dependent  functions  that  are 

performed  is  minimized; 

27.  Processing  data  volume/value  is  limited; 

28.  A  common,  standard  subset  of  a  programming 

language  is  used; 

If  developers  insured  that  the  system  exhibited  these 
traits  (specified  in  the  checklists),  then  they  would  be 
better  assured  of  achieving  a  more  reliable  and 
maintainable  system.  The  organizing  system  would  update 
the  AMT  database,  derive  the  metric  scores,  and  generate 
the  reports.  This  would  serve  as  a  more  quantitative  basis 

on  which  to  make  a  decision  about  the  progress  of  the 

system  development. 

At  this  point  it  became  infeasible  to  try  to  collect 
module  level  information  because  the  AMT  is  not  yet 
available  for  AFLC .  However,  if  a  checklist  is  provided  to 
programmers,  then  it  will  encourage  goal-directed 

programming.  Therefore,  the  software  is  more  apt  to 


exhibit  those  desired  quality  traits  (Ref  27).  Before 
coding,  programmers  should  be  given  a  checklist  specifying 
the  following  conditions  for  each  module  which  contributes 
to  reliable  and  maintainable  software: 


1.  Machine  level  language  statements  should  be 

minimized ; 

2.  Unconditional  branching  should  be  minimized; 

3.  A  specified  structured  language  will  be  used; 

4.  GOTO’s  should  be  avoided; 

5.  Prologue  comments  should  be  provided  which 

contain  information  about  the  function, 
author,  version  number,  date,  inputs, 
outputs,  assuiqptions ,  and  limitations  of 
the  modules; 

6.  There  should  be  a  camment  which  indicates  which 

itemized  requirement  is  satisfied  by  the 
modu le ; 

7.  All  decision  points  and  transfers  of  control 

should  be  commented; 

8.  Machine  language  code  must  be  commented; 

9.  All  non-standard  HOL  statements  must  be 

commented ; 

10.  All  declared  variables  must  be  described  by 

comments 

11.  All  variable  names  (mnemonics)  must  be 

descriptive  of  the  physical  or  functional 
property  they  represent; 

12.  Code  should  be  logically  blocked  and  indented; 

13.  No  line  should  contain  more  than  one  statement; 

14.  Inputs  should  be  range  tested; 

15.  Possible  conflicts  or  illegal  combinations  in 

inputs  should  be  checked; 

16.  Parameters  which  are  passed  to  or  from  other 

modules  must  be  defined  in  the  module; 
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17.  Global  variables  roust  be  used  consistently  with 

respect  to  units  or  type; 

18.  Bach  variable  should  be  used  for  one  purpose; 

19.  Loop  and  multiple  transfer  index  parameters  must 

be  range  tested  before  use; 

20.  Subscript  values  are  range  tested  before  use; 

21.  When  an  error  condition  occurs,  information  must 

be  passed  to  the  calling  module; 

22.  Results  of  a  computation  must  be  checked  before 

outputing  or  processing  continues; 

23.  During  execution,  outputs  must  be  within  accuracy 

tolerances. 

By  including  these  checklists  as  a  supplement  to 
development  procedures,  management  can  do  much  to  improve 
the  efficiency  and  effectiveness  of  the  SQA  program  that 
uses  the  AMT  because  these  checklists  correspond  to  the 
requirements  fur  the  metric  worksheets.  By  obtaining  item 
information  from  the  worksheets,  which  are  descriptive  in 
nature,  and  developing  the  checklists  to  be  used  before 
each  SDLC  phase,  SQM  becomes  more  prescriptive,  thus  aiding 
even  more  to  the  delivery  of  quality  software. 

Chapter  VII  summarizes  this  research  effort  and  makes 
recommendations  to  AFLC/LM  concerning  its  SQA  program. 
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CHAPTER  VII 


THESIS  RECOMMENDATIONS:  GUIDELINE  FOR  A  SQA  PROGRAM 


The  overall  objective  of  this  thesis  has  been  to 
provide  guidelines  to  AFLC/LM  concerning  the  way  Software 
Quality  Metrics  can  be  integrated  into  its  SQA  program. 

Research  Summary 

Chapter  I  provided  the  background  information  which 
identified  the  difficulties  AFLC  has  experienced  during  its 
attempts  to  establish  a  SQA  program.  It  pinpointed 
specific  symptoms  that  indicate  AFLC  lacks  a  viable  SQA 
function. 

In  Chapter  II,  Software  Quality  Metrics  were  presented 
as  a  means  to  provide  a  feasible  solution  to  this  problem. 
The  measurement  concepts  of  SQM  were  shown  to  complement 
other  software  tools  and  techniques,  and  to  identify 
quality  characteristics  that  these  other  aids  failed  to 
quantify.  The  chapter  focused  on  how  SQM  can  be  used  to 
identify  software  quality  requirements. 

Chapter  III  presented  the  methods  used  in  this  thesis 
to  derive  the  information  needed  to  make  recommendations 
concerning  the  development  of  AFLC/LM' s  SQA  program.  It 
outlined  the  research  methodology  required  to  demonstrate 
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the  feasibility  of  incorporating  Software  Quality  Metric.*: 
into  AFLC's  SQA  program. 

Chapter  IV  accomplished  the  first  goal  of  this  research 
efforts  to  determine  which  measurement  tools  and  techniques, 
should  be  used  by  AFLC/LM  in  establishing  its  SQA  program. 
It  concluded  that  the  minimum  set  of  tools  are:  the 
Automated  Measurement  Tool  (AMT),  Standards,  Standards; 
Analyzer,  Dynamic  Analyzer,  Language  Processor  and  a  Test; 
Bed,  and  that  the  minimum  set  of  techniques  should  be: 
Software  Quality  Metrics  (SQM),  Reviewing,  Auditing,  Design 
Inspections,  Walk-Throughs  and  Standardization.  By 
including  SQM  and  the  AMT  in  this  minimum  set  of  tools  and 
techniques  the  SQA  group  will  be  able  to  quantitatively 
assess  software  quality.  Chapter  IV  also  justified  the 
employment  of  a  toolsmith  within  the  SQA  organization. 

Chapter  V  proposed  a  systemic  model  of  the  software 
system  development  effort  that  can  be  used  to  conceptualize 
the  specific  information  requirements  of  the  subsystems 
within  the  phases  of  the  SDLC  and  thereby  accomplished  the 
second  goal  of  this  thesis:  to  develop  a  model  of  the 
decision-making  process  of  the  software  system  development 
effort  that  incorporates  the  application  of  SQM  and  which 
illustrates  the  feasibility  of  implementing  a 
quantitatively  oriented  SQA  program  at  AFLC/LM.  As  a 
result  of  this  modeling  effort,  software  development  has 
been  envisioned  as  a  controlled  management  process  in  which 
control  is  exercised  through  reviews,  status  reporting,  and 


121 


software  products  delivered  during  the  SDLC.  Currently, 
the  major  emphasis  of  the  control  function  is  to  evaluate 
the  schedule  and  cost  performance  and  to  determine  the 
functional  correctness  of  the  software  being  developed. 
The  concept  underlying  software  quality  metrics  is  to  use 
these  control  vehicles  to  provide  an  indication  ( anci 
therefore  a  mechanism  of  control)  of  the  quality  of  the 
software  product  to  be  delivered  (Ref  17). 

Chapter  VI  focused  on  accomplishing  the  third  goal:  to 
demonstrate  how  the  SQM  concepts  can  be  applied  at  AFLC/LM. 
It  achieved  this  by  providing  checklists  which  can  be  usee 
prior  to  each  phase  of  the  SDLC.  The  intent  of  the 
checklists  is  to  emphasize  the  fact  that  the  metrics,  which 
establish  the  quantitative  assessment  of  software  quality, 
are  based  on  the  availability  of  key  information  which 
exhibits  the  characteristics  of  the  software. 

It  has  been  found  that  the  costs  throughout  the  total 
life  cycle  are  more  affected  by  the  characteristics  of  the 
software  system  than  by  the  mission-oriented  functions 
performed  by  the  software  system  (Ref  35).  Large  software 
systems  have  sometimes  proven  untestable,  unmod  if iable ,  and 
largely  unusable  by  operations  personnel  because  of  the 
characteristics  of  the  software  (Ref  36). 

Research  Recommendations 

It  is  a  function  of  the  software  quality  assurance 
program  to  insure  that  the  characteristics  of  the  software 
system  satisfy  the  requirements  of  the  end-user.  However, 


by  solely  relying  upon  its  current  SQA  tools  and 
techniques,  AFLC/LM  cannot  provide  an  indication  of  the 
quality  of  the  software  it  delivers.  Therefore,  based  or 
the  findings  of  this  research  effort,  AFLC/LM  shoulc. 
incorporate  the  concepts  of  Software  Quality  Metrics  (SQM 
into  its  software  development  effort.  To  accomplish  this. 
AFLC  should: 

1.  Acquire  the  Automated  Measurement  Tool  (AMT)  which 
is  needed  to  operationalize  the  SQM  methodology.  To  obtain 
the  tool,  a  request  to  Joe  Cavano  (Autovon  587-7834)  at  the 


Rome  Air  Development  Center 

(RADC )  should 

be  made 

to 

acquire  the  software 

and 

tapes  for  the 

AMT. 

Also 

information  concerning 

implementation. 

testing 

and 

validation  should  be  acquired. 

2.  Develop  checklists,  similar  to  the  ones  in 
Chapter  VI,  to  be  used  by  software  developers  to  insure 
that  they  are  aware  of  the  software  traits  which  affect  all 
factors  of  software  quality.  These  checklists  should  be 
included  as  a  supplement  to  the  Project  Manager's  Handbook, 
which  is  now  being  developed  (Ref  89).  As  a  minimum,  the 
checklists  should  include  coverage  of  all  questions  in  the 
metric  worksheets  in  Appendix  A. 

3.  Contract  with  General  Electric's  Command  and 
Information  Systems  Division  at  Sunnyvale,  CA: 

a)  to  train  SQA  personnel  in  procedures  to  fully 
utilize  the  SQM  methodology  and  the  Automated  Measurement 


Tool  (AMT); 


b)  to  establish  standards  and  procedures  that  can 
be  used  by  the  SQA  program  in  assessing  the  quality  of 
software  products,  and 

c)  to  interface  existing  software  tools  at  AFLC  to 

the  AMT. 

4.  Assign  an  individual  in  the  SQA  program  to  be 
responsible  for  the  development  and  maintenance  of  written 
procedures  and  a  tool  library.  This  toolsmith  should  be 
familiar  with  other  software  tools  and  be  able  to  determine 
(through  procedures  in  Chapter  IV)  if  acquisition  of  the 
tool  is  justified. 

The  nature  of  the  framework  for  Software  Quality 
Metrics  allows  an  upper  level  application  of  the  SQM 
concepts  without  the  acquisition  of  the  AMT.  The 
structured  (step-wise)  procedures,  discussed  in  Chapter  II 
and  depicted  in  Figure  9,  provide  the  ability  to  specify 
system  requirements  in  software-oriented  terms.  This 
provides  goal-directed  design  and  implementation  which  has 
been  shown  to  improve  the  quality  of  the  end  product.  This 
fact  alone  means  AFLC/LM  now  has  the  capability  to 
significantly  impact  the  quality  of  its  software  systems 
before  development  begins. 

Once  it  acquires  the  AMT,  AFLC  can  expand  its 
capability  to  monitor  and  control  the  software  development 
effort.  The  AMT  will  serve  as  the  core  for  a  Decision 
Support  System  to  provide  management  with  an  objective 
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assessment  of  the  system  development.  By  integrating  other 
software  tools  with  the  AMT,  AFLC  can  effectively  automate 
the  monitoring  function  of  the  SDLC,  thus  reducing  the 
manpower  requirements  for  its  SQA  program. 

In  conclusion,  by  acquiring  the  Automated  Measurement 
Tool  to  support  the  application  of  Software  Quality 
Metrics,  AFLC/LM  will  significantly  improve  the  quality  of 
the  software  systems  it  develops  because  it  will  be  able  to 
quantitatively  measure  the  quality  of  the  software  system 
as  it  is  being  developed.  There  can  be  no  justification 
for  not  acquiring  this  assessment  capability. 
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APPENDIX  A 

METRIC  WORKSHEETS 


(Refs  17,25) 


T  H  l  C  WORKSHEET  1 

REQUIREMENTS  ANALYSIS/SYSTEM  LEVEL 


INSPECTOR. 


LOHPLETE.'.ESS  ( CORRECTNESS,  KEL  I  AS  I L  (  TV  ) 


1  Number  of  major  functions  identified  {equivalent  to  CPCI).  CP. I 

2  Are  requirements  itemited  so  that  the  various  functions  to  be  performed,  their 
inputs  and  outputs,  are  clearly  delineated?  CP. 1(1) 

3  Number  of  major  data  references  CP  1(2) 

4.  mow  many  of  these  data  references  are  not  defined?  CP. 1(2) 

5  How  many  defined  functions  are  not  used?  CP. I (3) 

6  How  many  referenced  functions  are  not  defined?  CP. 1(4) 

7.  How  many  data  references  are  not  used?  CP.  1(2) 

8.  How  many  referenced  data  references  are  not  defined?  CP. 1(6) 

9.  Is  the  flow  of  processing  and  all  decision  points  in  that  flow  described?  CP.  1(5 

10.  How  many  problem  reports  related  to  the  requi rements  have  been  recorded?  CP.  1(7) 

11  How  many  of  those  problem  reports  have  been  closed  (resolved)?  CP. I (7) 


II.  PRECISION  (RELIABILITY) 


1.  Has  an  error  analysis  been  performed  and  budgeted  to  functions?  AY. 1(1) 

2.  Are  there  definitive  statements  of  the  accuracy  requirements  for  inputs, 
outputs,  processing,  and  constants?  AY. 1(2) 

3.  Are  there  definitive  statements  of  the  error  tolerance  of  input  data?  ET.2(1) 

4.  Are  there  definitive  statements  of  the  requirements  for  recovery  from 
computational  failures?  ET.3(1) 

5.  Is  there  a  definitive  statement  of  the  requirement  for  recovery  from  hardware 
faults?  ET.4( 1 ) 

6.  Is  there  a  definitive  statement  of  the  requirements  for  recovery  from  device 
errors?  ET  .5(1) 


III  SECURITY  (INTEGRITY) 


1  Is  there  a  definitive  statement  of  the  requirements  for  user  input/output 
access  controls?  AC. 1(1) 

2  Is  there  a  definitive  statement  of  the  requirements  for  data  base  access 
controls’  AC. 1(2) 

3.  Is  there  a  definitive  statement  of  the  requirements  for  memory  protection 
a.  ross  tasks  1  AC.  1  (  3) 

4  Is  there  a  definitive  statement  of  the  requirements  for  recordinq  and 
reportinq  access  to  System?  AA . 1 ( 1 1 

5  1$  there  a  d“fimt1ve  statement  of  th  requirements  for  immediate 
indication  of  access  violation7  AA  .  I  , .  ) 
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IV  HUMAN  INTERFACE  (USABILITY  ) 


1.  Are  all  steps  in  the  operation  described  (operations  concept)?  OR. 1(1) 

2.  Are  ail  error  conditions  to  be  reported  to  operator/user  identified  and 
tne  responses  described?  OP. 1(2) 

3.  Is  there  a  statement  of  the  requirement  fur  the  capability  to  interrupt 
operation,  obtain  status,  modify,  and  continue  processing?  OP. 1(3) 

4.  Is  there  a  definitive  statement  of  requirements  for  optional  input  media? 

CM. 1(6) 

5.  Is  there  a  definitive  statement  of  requirements  for  optional  output  media? 

CM. 2(7) 

6.  Is  there  a  definitive  statement  of  requirements  for  selective  output 
control?  CM. 2(1) 


V.  PERFORMANCE  (EFFICIENCY) 


1.  Have  performance  requirements  (storage  and  run  time)  been  Identified  for 
the  functions  to  be  performed?  EE.l 


VI.  SYSTEM  INTERFACES  (INTEROPERABILITY) 


1.  Is  there  a  definitive  statement  of  the  requirements  for  communication  with 
other  systems?  CC.l(l) 

2.  Is  there  a  definitive  statement  of  the  requirements  for  standard  data 
representations  for  conmunlcation  with  other  systems?  DC. 1(1) 


VII.  INSPECTOR’S  COMMENTS 


Make  any  general  or  specific  coements  that  relate  to  the  quality  observed  while 
applying  this  checklist. 
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Is  there  a  matrix  relating  itemized  requirements  to  modules  winch  implement 
those  requirements?  TR.l 

How  many  major  functions  (CPCIS)  are  identified?  CP.  1 
How  many  functions  identified  are  not  defined?  CP. 1(2) 

How  many  defined  functions  are  not  used?  CP. 1(3) 

How  many  Interfaces  between  functions  are  not  defined?  CP. 1(6) 

Humber  of  total  problem  reports  recorded?  CP. 1(7) 

Number  of  those  reports  that  have  not  been  closed  (resolved?)  CP. 1(7) 


Profile  of  problem  reports:  (nuaiber  of  following  types)  •-  Computational 

b.  Logic 

II.  PRECISION  (RELIABILITY) _ ( _ _  ^  oS'taSffng 

Have  math  library  routines  to  be  used  been  l"”~ "T””  OS/System  Support 

f.  Configuration 

checked  for  sufficiency  with  regards  to  Y  H  g.  Routine/Soutine 

accuracy  requirements?  AY. 1(3)  interface 

_  h.  Routine/ System 

Is  concurrent  processing  centrally  Interface 

controlled?  El.  1,1)  J_1  ]:  ffiSSST 

How  many  error  conditions  are  reported  k.  data  base  Interface 

by  the  syitml  IT. 1(2)  *  | 

How  many  of  those  errors  ere  automatically  m.  Preset  data 

fixed  or  bypassed  and  processing  continues? 

How  many,  require  operator  Intervene^ j*  p  Recurrent  errors 

Are  provisions  for  recovery  from  hardware  ~  r  q  Documentation 

faults  provided?  ET.4(2)  r.  Requirement 

Are  provisions  for  recovery  from  device  compliance 

errors  provided?  ET.5(2)  Y  N  \  Suestions 

——————————————————————— - - I——1  <  u.  Hardware _ 

III  STRUCTURE  (RELIABILITY,  MAINTAiNABI L l TY  .1ESABILITY , 

PORTABILITY,  REUSABILITY,  INTEROPERABILITY) 

Is  a  hierarchy  of  system,  identifying  all  modules  in  the  system  provided? 

Number  of  Modules  51.1(2)  SI. 1(1) 

Are  there  any  duplicate  funttions7  SI. 1(2) 

Based  on  hierarchy  or  a  call/called  matrix,  how  many  modules  are  called  by 
more  than  one  ither  module?  OE  .  1 

A  re  the  .in  r  n  1 1 \  u'.ed  in  r  he  system  define)  imi  e  '  CE  2  (  5 ) 
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iv  optimization  (tfncitNCY) 

1.  Are  storage  requirements  allocated  to  design?  SE . I ( 1 ) 

2  Are  virtual  storage  facilities  used?  SE .1(2) 

Is  dynamic  memory  management  used?  SE.I(S) 

A.  Is  a  performance  optimizing  compiler  used?  SE .1(7) 

5.  Is  global  data  defined  once?  CS.2(T) 

6.  Have  Data  Base  or  files  been  organized  for  efficient  processing?  EE. 3(5) 

7.  Is  data  packing  used?  EE. 2(5) 

8.  Number  of  overlays  E£.2(4) 

9.  Overlay  efficiency  -  memory  allocation  EE. 2(4) 

max  overlay  size 

min  overlay  size 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

V 

N 

V 

N 

Y 

N 

V.  SECURITY  (INTEGRITY) 

1.  Are  user  Input/Output  access  controls  provided?  AC. 1(1) 

2.  Are  Oata  Base  access  controls  provided?  AC. 1(2) 

3.  Is  memory  protection  across  tasks  provided?  AC. 1(3) 

4.  Are  there  provisions  for  recording  and  reporting  errors?  AC. 2(1 ,2) 

Y 

N 

Y 

N 

V 

N 

Y 

N 

VI.  SYSTEM  INTERFACES  (INTEROPERABILITY) 

1.  Now  many  other  systems  will  this  system  Interface  with?  CC.!(1) 

2.  Have  protoc  1  standards  been  established?  CC.1(2) 

3.  Are  they  being  complied  with?  CC .3(2) 

4.  Number  of  modules  used  for  Input  and  output  to  other  systems?  CC. 1(3,4) 

5.  Has  a  standard  data  representation  been  established  or  translation 
standards  between  representations  been  established?  OC. 1(1) 

6.  Are  they  being  complied  with?  DC. 1(2) 

7.  Number  of  modules  used  to  perform  translations?  0C.1(3) 

1 

Y 

K  . 

Y 

N 

r 

N 

Y 

N 

VII.  HUMAN  INTERFACE  (USAfcILlTY) 

1.  Are  all  steps  in  operation  described  including  alternative  flows?  OP. 1(1) 

2.  Number  of  operator  actions?  OP  1(4) 

Ur 

LnJ 

METRIC  WORKSHEET  2a 
OESIGN/SYSTEM  LEVEL 


SYSTEM 

NAME: 


OATE. 

INSPECTOR 


T9  3 


1 


1  _ 1 - 1 - - 

VII.  HUMAN  INTERFACE  (USABILITY)  Continued 

3. 

Estimated  or  Actual  time  to  perform?  OP. 1(4) 

4. 

Budgeted  time  for  complete  job?  OP. 1(4) 

Are  job  set  up  and  tear  down  procedures  described?  OP. 1(5) 

S. 

6. 

Is  a  hard  copy  of  operator  Interactions  to  be  maintained?  OP. 1(6) 

Y 

7. 

Number  of  operator  messages  and  responses?  OP. 1(2) 

8. 

Number  of  different  formats?  OP. 1(2) 

9. 

Are  all  error  conditions  and  responses  appropriately  described?  OP. 1(2) 

10. 

Ooes  the  capability  exist  for  the  operator  to  Interrupt,  obtain  status, 
save,  modify,  and  continue  processing?  OP. 1(3) 

Y 

■ 

11. 

Are  lesson  plans/tralnlng  materials  for  operators,  end  users,  and 
maintainors  provided?  TN.l(l) 

Y 

II 

12. 

Are  realistic,  simulated  exercises  provided?  TN.1(2) 

Y 

N 

13. 

Are  help  and  diagnostic  Information  available?  TH .1(3) 

14. 

Number  of  Input  formats  CM.1(2) 

15. 

Number  of  Input  values  CM.  1(1) 

16. 

Number  of  default  values  CM. 1(1) 

17. 

Number  of  self- Identifying  Input  values  CM.  1(3) 

18. 

Can  Input  be  verified  by  user  prior  to  execution?  CM. 1(4) 

19. 

Is  Input  terminated  by  explicitly  defined  by  logical  end  of  input?  CM. 1(5 

J L 

M 

20. 

Can  Input  be  specified  from  different  media?  CM. 1(6) 

r 

N 

21. 

Are  there  selective  output  controls?  CM. 2(1) 

r 

N 

22. 

23. 

24. 

25. 

26 

27. 

Do  outputs  have  unique  descriptive  user  oriented  labels?  CM. 2(5) 

Do  outputs  have  user  oriented  units?  CM. 2(3) 

Number  of  output  formats?  CM.2(4) 

Are  logical  groups  of  output  sepanted  for  user  emamination7  CM. 2(5) 

A *e  relationships  between  error  messages  and  outputs  unambiguous7  CM. 2(6] 
Are  there  provisions  fo^  directin')  output  to  different  media7  CM. 2(7) 

_ Y 

_  Y 

r 

Y 

Y 

N  _ 
N_ 

.1. 

N_ 

N 

VlH  TESTING  (TESTABILITY)  APPLY  TO  TEST  P|  AN .  PROCEOUBf  S ,  RESULTS 

■■  ■  ~  '  ,  11  i  —  t  . —  "  - -r-  -  ■■ 

I.  Number  of  piths?  IN.  1(1) 

4  Number  of  input  t  » 

--i 

2.  Number  of  pa*hs  to  be  tested? 

be  tested’  JN  i ' ) 

>  _ _ 

IN . 1 ( l  , 

3.  Number  of  moot  parameters7 

S  Numb*’ ■  of  ir'tbi  **s  1  ?(  * 

M  i  ; 

METRIC  IhOPHSmLE T  _'j 

SYSTEM 

OATE. 

JE SIGN/ SYSTEM  LEVEL 

NAME  ; 

INSPECTOR; 

VIII.  TESTING  (TESTABILITY)  -  APPLY  TO  TEST  PLAN,  PROCEDURES.  RESULTS  (CONTINUED) 


6  Number  of  interfaces  to  be  tested’  IN. 2(1) 

9  Number  of  modules?  IN. 1(1) 

— 1 

7  Number  of  itemized  performance  requirements? 

jm 

10.  Number  of  modules  to  be 

■  I 

»  IN.2(?/ 

exercised?  IN.  3(1) 

tested?  IN. 2(2) 

<1  Are  test  inputs  and  outputs 

B 

provided  in  summary  fgrsj^ 

1 X  OATA  BASE 


1  Number  of  unique  data  Item  in  data  base  SI. I (6) 

2.  Number  of  preset  data  items  SI .  1(6) 

3.  Ngafeer  of  major  screen  ts  (files)  in  data  base  SI. 1(7) 

*  INSPECTOR'S  COWCNTS 


Hake  any  general  or  specific  caamwnts  about  the  quality  observed  uhilr  applying  this 
checklist. 
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f.U* 


METRIC  wOPxbH.  ET  2 j 

SYSTEM 

DATE. 

DESICN/SYSTEN  LEVEL 

NAME  . 

INSPECTOR: 

_ 

VIII.  TESTING  (TESTABILITY) 

-  APPLY  TO  TEST  PLAN.  PROCEDURES. 

RESULTS  (CONTINUED) 

6. 

Number  of 

interfaces  to  be  tested’ 

IN. 2(1 ) 

7 

Number  of 

itemized  performance  requi  remeiHs’ 

9 

Number  of 

performance  requirements 

to  be 

tested? 

IN. 2(2) 

9  Number  of  modules?  IN. 3(1) 

10.  Number  of  modules  to  be 
enercised?  IN.  3(1) 

II  Are  test  inputs  and  outputs 
provided  In  suMMry  form? 


\  mi 


IX  OATA  BASE 


1.  Number  of  unique  data  items  in  data  base  SI. 1(6) 

2.  Number  of  preset  data  items  SI. 1(6) 

3.  Number  of  major  segments  (files)  in  data  base  SI.  1(7) 


X  INSPECTOR’S  CONSENTS 


METRIC  WORKSHEET  2b 
OESIGN/MOOULE  LEVEL 

SYSTEM  NAME : 

OATE: 

NODULE  NAME: 

INSPECTOR: 

I.  COMPLETENESS  (CORRECTNESS,  RELIABILITY)  j 

1.  Can  you  clearly  distinguish  inputs,  outputs,  and  the  function  being  performed?  CP.  l  ( I  >| 

2.  How  many  data  rtfannees  are  not  defined,  computed,  or  obtained  from  an  external 
source?  CP. 1(2) 

3.  Are  all  conditions  and  processing  defined  for  each  decision  point?  CP. 1(5) 

4.  Now  many  problem  reports  have  been  recorded  for  this  module?  CP. 1(7) 

a.  Computational 


5.  Profile  of  Problem  Reports: 

6.  Humber  of  problem  reports  still  outstanding  CP.1(7]J~ 


II.  PRECISION  (RELIABILITY) 


1.  When  an  error  condition  is  detected,  is  it 
passed  to  calling  module?  ET.1(3) 

2.  Have  numerical  techniques  being  used  in  algori¬ 
thm  been  analysed  with  regards  to  accuracy 
requirements?  AT. 1(4) 

3.  Are  values  of  inputs  range  tested?  ET.2(2) 

4.  Are  conflicting  requests  and  illegal  combina¬ 
tions  identified  and  checked?  ET.2(3) 


5.  Is  there  a  check  to  see  if  all  necessary  data 

is  available  before  processing  begins?  ET.2(5)  I 


Is  all  input  checked,  reporting  all  errors, 
before  processing  begins?  ET.2(4) 


7.  Are  loop  and  multiple  transfer  Index  parameters 
range  tested  before  use?  ET.3(2) 


8.  Are  subscripts  range  tested  before  use’  ET.3(3) 

9.  Are  outputs  checked  for  reasonableness  before 
processing  continues?  ET.3(4) 


-I 


b.  Logic 

c.  Input/Output 

e.  System/OS  Support 

f.  Configuration 

g.  Routine/ Routine  Inter, 
face 

h.  Routine/System  Inter¬ 
face 

I.  Tope  Processing 

J.  User  Interface 

k.  Oata  Base  Interface 

l.  User  Reouested  Changes 

"v  Preset  Oata 

n.  Global  Variable  De’1 
nl tlon 

P.  Recurrent  Errors 
')  Oocumenlat  Ion 
r  Requirement  Cornpl'ame 
*  Operator 
u  Questions 
v.  hardware 


V  N 


III.  STRUCTURE  (REt  [ABILITY,  HA  IhTAtNAe  !U  Tv  ,  TFSTAr  Ml  t  v  ) 


1 


How  many  Oecislqn  Points  are  them’ 

SI  .  3 


2.  How  many  subdecision  Points  am 
there’  SI.  3 


*1ow  mar/  r  omt  1 1  1  on*  1  bran 

•he,*'  SI  ) 

Mow  I'M'?  .  -i»'  <  ond*  t  » <~*Ma  1 

()fp  IhpfO  f  S|  | 


anrhes 
hranr  h. 
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Copy  av..u_ol^  to  DTIC  does  not 
peunit  fully  legible  reproduction 


I 


ICTRlt  WORKSHEET  ?B 

design /module  level 


SYSTEM  NAME: 
MOOULE  NAME: 


III.  STRUCTURE  (RELIAeil  ;TY,  MAlNI  A1NAB  It  1 1  Y ,  TEST  AB I L I T  Y )  (CONTINUED) 

S  Is  the  module  dependent  on  the  [  7.  Are  any  limitation 

source  of  the  input  or  the  I  sing  performed  by 

destination  of  the  output?  SI. 1(3)  |  1 2 3  identified?  EX. 21 

6.  Is  the  module  dependent  on  know-  •  8.  Number  of  entranci 


IV  REFERENCES  (MAINTAINABILITY,  FLEXIBILITY,  TESTABILITY.  PORTABILITY.  REUSABILITY . 
INTEROPERABILITY) 


r 

i-V 

ti 

7. 

Are  any  limitations  of  the  proces¬ 
sing  performed  by  the  module 

Identified?  EX. 2(1) 

U- 

N 

8. 

Number  of  entrances  into  m^uj^ 

r 

9. 

Number  of  exits  from  module 

ale  1  \  9 | 

1.  Number  of  references  to  system 

8.  Is  temporary  storage  shared  with 

library  routines,  utilities  or  other 

other  modules?  MO. 2(7) 

system  provided  facilities  SS.I(1) 

9.  Does  the  module  mix  Input,  out- 

2.  Number  of  Input/output  actions 

put  and  processing  functions  In 

MI. 1(2) 

— 

same  nodule?  GE.2(1) 

3.  Number  of  calling  sequence  parameters 

MO. 2(3) 

— 

— 

10.  Nuafcer  of  machine  dependent 

4.  How  many  calling  sequence  parameters 

functions  performed?  GE.2(2) 

are  control  variables  MO. 2(3) 

11.  Is  processing  data  volume  1 1ni tedl 

5.  Is  input  passed  as  calling  sequence 

3) 

parameters  NO. 2(4) 

Y 

N_ 

12.  Is  processing  data  value  Hnl^? 

6.  Is  output  passed  back  to  cal’ing 

13.  Is  a  comaon,  standard  subset  of 

module?  MO. 2(5) 

Y 

N 

prograaaelng  language  to  be  used7 

SS.1(2) 

7.  Is  control  returned  to  calling 

14.  Is  the  programing  language 

module  MO. 2(6) 

available  In  other  machines? 

. 

_ 

MI. 1(1) 

V.  EXPANDABILITY  (FLEXIBILITY) 

1.  Is  logical  processing  independent  of  storage  specification?  EX. 1(1) 

2.  Are  accuracy,  convergence,  or  timing  attributes  parametric?  EX. 2(1) 

3.  Is  module  table  driven?  EX. 2(2) 

VI  OPTIMIZATION  (EFFICIENCY) 

1.  Are  specific  perfjrmance  requirements  (storage  and  routine)  allocated  to  this 
module?  EE.l 
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METRIC  WORKSHEET  2b 
DESIGN /MODULE  LEVEL 


SYSTEM  NAME: 
MODULE  NAME 


VI.  OPTIMIZATION  (EFFICIENCY)  (CONTINUED) 


2.  Which  category  does  processing  fall  in:  EE. 2 

Real- t ime 
On-line 

T ime-constrained 
Non-time  critical 

Are  non-loop  dependent  functions  kept  out  of  loops?  EE.2(i) 
Is  bit/byte  packing/unpacking  performed  in  loops?  EE.2(S) 

S.  Is  data  Indexed  or  reference  efficiently?  EE.  3(5) 


VII.  FUNCTIONAL  CATEGORIZATION 


Categorize  function  performed  by  this  module  according  to  following: 


□ 

□ 

□ 

□ 

□ 

□ 


CONTROL  -  an  executive  module  whose  prime  function  is  to  invoke  other  modules. 


INPUT/OUTPUT  •  a  module  whose  prime  function  is  to  communicate  data  between 
the  computer  and  the  user. 

PRE/POSTPROCESSOR  -  a  module  tdiose  prime  function  is  to  prepare  data  for  or 
after  the  invocation  of  a  computation  or  data  management 
module. 

ALGORITHM  -  a  module  whose  prime  function  Is  computation. 


DATA  MANAGEMENT  -  a  moudle  whose  prime  function  Is  to  control  the  flow  of 
data  within  the  computer. 

SYSTEM  -  a  module  whose  function  is  the  scheduling  of  system  resources  for 
other  modules. 


VIII .  COIiSISTEilCY 


1  Does  the  design  representat ion  comply  with  established  standards  CS.l(l) 

2.  Do  input.'output  references  comply  with  established  standards  CS.K3) 

3.  Do  calling  sequences  comply  with  established  standards  CS.1(?) 

4.  Is  error  handling  done  according  to  established  standards  CS.M4) 

5  Are  variable  named  according  to  established  standards  CS  2(7) 

6  Are  global  variables  used  as  defined  globally  CS.2(3) 


148 


METRIC  WORKSHEET  2t» 
DESIGN/ MODULE  LEVEL 


SVSUN  NAME : 
MODULE  NAME ; 


Po.  4 


IK.  INSPECTOR'S  COMMENTS 

Make  any  specific  or  general  comments  about  the  quality  observed  while  applying  this 
check  I ist? 


NITRIC  WORKSHEET  3 

SYSTEM  NAME : 

DATE: 

SOURCE  COOE/NOOULE  LEVEL 

MODULE  NAfC: 

INSPECTOR: 

I.  STRUCTURE  (RELIABILITY, 

,  MAINTAINABILITY,  TESTABILITY) 

Number  of  lines  of  code  MO. 2(2) 

Number  of  lines  excluding  consents 

SI. 4(2) 

Number  of  Machine  level  language 
statements  SO. 3(1) 

Number  of  declarative  state«ents 

SI. 4 

Nwber  of  data  Manipulation  state* 
Mints  SI. 4 

Hmbor  of  sta tenant  labels  $1  «($> 
(Oo  not  count  fornat  statenents)' 
Rimber  of  entrances  Into  module 

SI. 1(5) 

Nueber  of  exits  from  Module 

SI.1(5) 

MOxImum  nesting  level  SI. 4(7) 

Nueber  of  decision  points 

(IF.  WHILE.  REPEAT,  DO.  CASE)  SI. 3 


11.  Nueber  of  sub-decision  points 

SI. 3 

12.  Number  of  conditional  branches 
(computed  go  to)  SI .4(8) 

13.  Number  of  unconditional  branches 
(GOTO.  ESCAPE)  SI. 4(9) 

14.  Number  of  loops  (WHILE.  DO) 

SI. 4(3) 

15.  Nueber  of  loops  with  jumps  out  of 
loop  SI. 4(3) 

16.  Nueber  of  loop  Indicias  that  are 
modified  SI. 4(4) 

17.  Nueber  of  constructs  that  perform 
module  modification  (SWITCH, 
ALTER)  SI.  4(5) 

18.  Nueber  of  negative  or  comnllcated 
compound  boolean  expressions 

19.  Is  a  structured  language 

SI. 2 

20.  Is  flow  top  to  bottom  (are  there 
any  backward  branching  60TOs)sl 


CONCISENESS  (MAINTAINABILITY)  •  SEE  SUPPLEMENT 


Nueber  of  operators  C0.1 
Nueber  of  unique  operators  CO. I 


3.  Ntmber  of  Operands  C0.1 

Nimber  of  unique  operands  C0.1 


III.  SELF- OESCR I PT I VENESS  (MAINTAINABILITY,  FLEXIBILITY,  TESTABILITY,  PORTABILITY.  REUSABILITY) 


1.  Nueber  of  lines  of  coeseents  SD.l  7.  Are  non-standard  HOI  statements 

-  commented?  SD.2(5)  - 1 - 

2.  Nueber  of  non -blank  lines  of  coeseents  Y  N 


Nueber  of  non-blank  lines  of  coeseents 

SD.l 

Are  there  prologue  coeseents  provided 
containing  Information  about  the 
function,  author,  version  number, 
date.  Inputs,  outputs,  assumptions 
and  limitations? 

Is  there  a  comment  which  Indicates 
what  Itemized  requirement  is 
satisfied  by  this  module?  SO .2(1) 

Mow  many  decision  points  and  trans¬ 
fers  of  control  are  not  cotimented? 

SO. ?( 3) 

Is  all  machine  lanquaqe  code  com 
mented?  SO. 7(4) 


How  many  declared  variables  are 
not  described  by  comments? 

SD . 2(6) 

Are  variablp  names  (mnemonics) 
descriptive  of  the  nhysicai  or 
functional  property  they 
rep  re  .ent  ’  SO .  7) 

Do  the  ronwents  do  more  than 
repeat  'he  operation’  SO. 2(7' 


Is  the  code  loqiia’lv  blocked  and 
indented  ’  SO.  1(  1) 


Number  I*  lines  v’d  m ore  than 
I  st  a t  emer t  '>0  '( t ) 


in 

IF 

TF" 


Nwmh*  *  ii»  .  ,  in  f  •  »>  i 


ni’  I  *  f’»*^ 

•0  Ml) 
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METRIC  WORKSHEET  1 
SOURCE  CODE >HOOULE  LEVEL 


SYSTEM  NAME 
MODULE  NAME 


Pg.  2 


IV 

INPUT/CHITPUT  (RELIABILITY, 

FLEXIBILITY. 

PORTABILITY) 

□ 

1 

Number  of  input  .tateinents 

MI  .1(2) 

4 

Are  inputs  range-tested  (for 
inputs  via  calling  sequences, 
global  data,  and  Input 
statements)  ET .2(2) 

Are  possible  conflicts  or  illegal 
combinations  in  fnputs  checked? 

ET.2{3) 

2. 

Number  of  output  statements 

Ml. 1(2) 

i' 

3. 

Is  amount  ot  input  that  can 
handted  parametric?  G£.2(J) 

be 

5 

Y 

1“ 

Y 

6 

Is  there  a  check  to  determine  if 
all  data  is  available  prior  to 
processing?  ET.2(5) 

_ 

□ 

□ 

V.  REFERENCES  (RELIABILITY.  MAINTAINABILITY.  TESTABILITY.  FLEXIBILITY.  PORTABILITY.  REUSABILITY) 


1.  Nueber  of  calls  to  other  modules 

HO. 2(1) 

2.  Number  of  references  to  system 
library  routines,  utilities,  or 
other  system  provided  functions 

SS.I(l) 

3.  Number  of  calling  sequence  parameters 

MO. 2(3) 

4.  How  many  elements  In  calling 
sequences  are  not  parameters  ? 

HD. 2(3) 

5.  How  deny  of  the  calling  parameters 
(Input)  arc  control  variables? 

MO. 2(3) 


How  many  parameters  passed  to  or 
from  other  modules  are  not  defined 
In  this  module?  MO. 2(3) 

Is  Input  data  passed  as  parameter? 

NO. 2(4) 


Is  output  data  passed  back  to 
calling  module?  MD.2(S) 


Is  control  returned  to  calling 
nodule?  MO. 2(6) 


VI.  DATA  (CORRECTNESS.  RELIABILITY,  MAINTAINABILITY,  TESTABILITY) 


1.  Nueber  of  local  variables  SI. 4(10) 

2.  Nuafcer  of  global  variables  SI. 4(10) 

3.  Number  of  global  variables  renmnad 

SE.1(3) 


How  many  global  variables  are  not 
used  consistently  with  respect  to 
units  or  type?  cS. 2(4) 

How  many  variables  are  used  for 
more  than  one  purpose?  CS.2(3) 


VII.  ERROR  HANOLING  -  (RELIABILITY) 


VIII.  (EFFICIENCY) 


1.  How  many  loop  and  multiple  transfer 
Index  parameters  are  not  range 
tested  before  use?  ET.3(2) 

2.  Are  subscript  values  range  tested 
before  use?  ET.3(3) 

3.  When  an  error  condition  occurs.  Is  It 
passed  to  the  calling  module?  ET.|(J) 

4.  Are  the  results  of  a  computation 
checked  before  outputting  or  before 
processing  continues?  ET  1(4) 


Y  N 


7TV 


1.  Number  of  mix  mode  expressions? 

EE.  3(3) 

2.  How  many  variables  are  Initial  Tied 
when  declared?  EE. 3(2) 

3.  How  many  loops  have  non-loop 
dependent  statements  In  them? EE. 20 

4.  How  many  loops  have  b<t/byte 
packing/unpacking?  EE. 2(5) 

SE. 1 (6) 

5.  How  many  compound  expressions 
defined  more  than  once?  EE. 2(3) 
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METRIC  WORKSHEET  3 
SOURCE  COOE/MOOULE  LEVEL 


SYSTEM  NAME: 
MODULE  NAME. 


Pg.  3 


J. i- 


IX. 

portability 

X. 

FLEXIBILITY 

1. 

Is  code  Independent  of  word  and 
character  site?  MI. 1(3) 

i . 

Is  module  table  driven  EX.2(?) 

B 

O 

2 

Are  there  any  limits  to  data 
values  that  can  be  processed? 

GE.2(4) 

2. 

Nuefeer  of  lines  of  machine  language 
stataamnts.  MX .  1 

Is  data  representation  machine 
Independent?  MI. 1(4) 

3. 

3. 

Are  there  any  limits  to  amounts 
of  data  that  can  be  processed? 

0E.2( 3) 

B 

O 

4. 

Is  data  access/storage  system  soft¬ 
ware  Independent?  SS.1 

4. 

Are  accuracy,  convergence  and 
timing  attributes  parametric? 

EX. 2(1) 

B 

O 

XI.  DYNAMIC  MEASUREMENTS  (EFFICIENCY,  RELIABILITY) 


1.  During  execution  are  outputs  within  accuracy  to)erance*7  AY. 1(5) 

2.  During  andule/development  tasting,  what  was  run  tint?  EX. 2(3) 

3.  Complete  memory  map  for  execution  of  this  module  SE.1(4) 

_ _ _ Site  (words  of  memory) 


APPLICATION 

SYSTEM 

DATA 

OTHER 

4.  During  execution  how  many  data  1 tarns  ware  referenced  but  not  modified  EE. 3(6) 

5.  During  execution  how  many  data  Items  ware  modified  EE. 3(7) 


XII.  INSRECTORS  COMMENTS 


Make  any  general  or  specific  coements  that  relate  to  the  quality  observed  by  you  while 
applying  this  checklist: 


APPENDIX  B 

WORKSHEET  REQUIREMENTS 
(Ref  51) 


WORKSHEET  REQUIREMENTS 


The  following  is  a  description  of  the  data  that  the 
Automated  Measurement  Tool  (AMT)  requiresf/v  f  17,23).  The 
following  conventions  apply  to  certain  documents  and  source 
code : 

As  an  overview: 

o  Worksheet  1  is  manually  applied  to  the  requirements 
analysis  at  the  system  level. 

o  Worksheet  2a  is  manually  applied  to  the  design  docu¬ 
ment  at  the  system  level. 

o  Worksheet  2b  is  manually  applied  as  required  to  the 
design  documents  at  the  module  level. 

o  Worksheet  3  is  applied  to  the  code.  Many  of  the 
values  will  be  collected  automatically  by  AMT  and 
entered  into  the  data  base  for  further  usage.  The 
remaining  items  will  have  to  be  entered  manually. 

All  data  may  be  included  in  the  AMT  database  provided  the 
following  conventions  are  to  be  used  in  filling  out  the 
worksheets : 

1.  If  a  numerical  value  is  called  for,  enter  a  real 
number  of  -1  for  not  applicable  responses. 

2.  If  a  "yes"  or  "no"  response  is  called  for,  circle 
"Y"  or  "N".  For  not  applicable  responses  write  NA 
over  the  Y  or  N. 

3.  If  the  item  is  numeric  and  is  not  applied  to  the 
development  in  question,  e.g.,  the  development 
standards  do  not  require  problem  reports  to  be 
filed  against  the  requirements,  or  otherwise  "not 
applicable"  enter  a  negative  (-1)  or  "NA". 

4.  The  name  of  the  system  and/or  module  must  be 
common  across  all  worksheets. 

5.  A  separate  worksheet  number  2b  and  1  must  r>e  filled 
out  for  each  module. 


6.  A  maximum  of  180  characters  may  be  entered  into  the 
comments  section  of  each  worksheet. 

The  data  elements  are  defined  below  in  outline  form 
corresponding  to  the  worksheets.  Each  worksheet  is 
self-contained  and  uses  no  raw  data  beyond  itself. 


WORKSHEET  1 


This  worksheet  will  use  the  requirements  document (s)  and  an 
analyst  will  manually  fill  out  the  worksheets. 

I.  COMPLETENESS 

1.  Self  explanatory 

2.  Self  explanatory 

3.  Self  explanatory 

4.  Refers  to  1-3 

5.  Refers  to  1-1 

6.  Refers  to  1-1 

7.  Refers  to  1-3 

8.  Refers  to  1-4  and  1-7 

9.  Self  explanatory 

10.  Self  explanatory,  often  not  applicable.  If 
NA,  enter  -1. 

11.  Refers  to  1-10.  If  1-10  is  NA  this  must  be  -1. 
II.  PRECISION 

1-6.  All  are  yes  or  no  responses 

III.  SECURITY 

1-5.  All  are  yes  or  no  responses 

IV.  HUMAN  INTERFACE 

1-6.  All  are  yes  or  no  responses 

V.  PERFORMANCE 

1.  May  be  stated  in  either  numbers  or  prose 

VI.  SYSTEM  INTERFACES 
1.  yes  or  no 

VII.  INSPECTOR’S  COMMENTS 

Free  format,  limited  to  180  characters 


WORKSHEET  2a 

This  worksheet  is  used  to  evaluate  the  design  documents  at 
the  system  level  and  an  inspector  will  manually  fill  out 
one  for  the  system  design. 


I. 


COMPLETENESS 

1.  Mandatory  yes  or  no 

2.  Self  explanatory 


3- 5.  Refer  to  1-2 

6.  May  be  NA.  If  NA,  enter  -1. 

7-27.  If  1-6  is  NA,  all  must  be  -1. 

II.  PRECISION 

1-2.  Mandatory  yes  or  no 

3.  Requires  a  positive  integer  other  than  zero 

4- 5.  Zero  or  positive  integer 

6- 7.  Mandatory  yes  or  no 

III.  STRUCTURE 

1-2.  Mandatory  yes  or  no 

3.  Requires  a  positive  integer  other  than  zero 

4.  Zero  or  positive  integer 

5.  Mandatory  yes  or  no 

IV.  OPTIMIZATION 

1-5.  Mandatory  yes  or  no 

6.  Yes,  No,  or  NA 

7.  Mandatory  yes  or  no 

8.  Positive  integer  or  zero 

9-11.  If  IV- 8  is  zero,  must  be  zero  or  -1. 

V.  SECURITY 

1.  Mandatory  yes  or  no 

2.  Yes,  No,  or  NA 

3-4.  Mandatory  yes  or  no 

VI.  SYSTEM  INTERFACES 

1.  Positive  integer  or  zero 

2.  Mandatory  yes  or  no 

3.  Yes  or  no.  If  VI -2  is  No,  must  be  No. 

4.  Positive  integer  excluding  zero 

5.  Mandatory  yes  or  no 

6.  If  VI-5  is  no,  must  be  no 

7.  Positive  Integer  or  zero 

VII.  HUMAN  INTERFACE 

1.  Mandatory  yes  or  no 

2.  Positive  integer  excluding  zero 

3-4.  Either  zeros  or  a  unit  figure.  Be  sure 
measurement  units  are  same  for  3  and  4. 

5- 6.  Mandatory  yes  or  no 

7- 8.  Positive  integer  or  zero 

9-13.  Mandatory  yes  or  no 

14-17.  Positive  integer  or  zero 

18-23.  Mandatory  yes  or  no 

24.  Positive  integer  or  zero 

25-27.  Mandatory  yes  or  no 

VII E.  TESTING 

1-10.  Positive  integers  or  zero 
11.  Mandatory  yes  or  no 


IX.  DATA  BASE 

1-3.  Positive  integers  or  NA 

X.  INSPECTOR ' S  COMMENTS 

Free  format,  limited  to  180  characters. 


WORKSHEET  2b 

This  worksheet  is  used  to  evaluate  the  design  documents  at 
the  module  leve.  An  inspector  will  fill  out  one  for  each 
module  design  document ( s ) . 

I.  COMPLETENESS 

1.  Mandatory  yes  or  no 

2.  Positive  integer  or  zero 

3.  Mandatory  yes  or  no 

4- 5.  Positive  integers,  zero,  or  NA.  If  NA, 

enter  -1. 

6-24.  Positive  integers,  zero,  or  NA.  If  1-4  is 
NA,  (-1)  6-24  is  NA.  All  NA's  are  entered 
as  -1. 

II.  PRECISION 

1-9.  Mandatory  yes  or  no 

III.  STRUCTURE 

1-4.  Positive  integers  or  zero 

5- 7.  Mandatory  ye.-,  or  no 

8-9.  Positive  integer  excluding  zero 

IV.  REFERENCES 

1-4.  Positive  integers  or  zero 
5-14.  Mandatory  yes  or  no 

V.  EXPANDABILITY 

1-3.  Mandatory  yes  or  no 

VI.  OPTIMIZATION 

1.  Mandatory  yes  or  no 

2.  Circle  1,  2,  3,  or  4 
3-5.  Mandatory  yes  or  no 

VII.  FUNCTIONAL  CATEGORIZATION 
Circle  1,  2,  3,  4,  5,  or  6 

VIII.  CONSISTENCY 

1-6.  Mandatory  ye*  or  no 

IX.  INSPECTOR'S  COMMENT.’- 

Free  format,  limited  to  180  characters. 


WORKSHEET  3 


This  worksheet  is  used  to  evaluate  the  source  code  of  each 
module.  An  inspector  must  enter  the  following  items 
manually.  The  remainder  are  automatically  gathered  from 
the  source  code. 

I.  STRUCTURE 

3,  14,  15,  16,  and  18  Positive  integer  or  zero 


19-20 

.  Mandatory  yes  or 

no 

• 

H 

H 

CONCISENESS 

1-4. 

Positive  integer 

m. 

SELF- 

DESCRIPTIVENESS 

2. 

Positive  integer 

or 

zero 

3-4. 

Mandatory  yes  or 

no 

5. 

Positive  integer 

or 

zero 

6-7. 

Yes,  no,  or  NA 

8. 

Positive  integer 

or 

zero 

9-11. 

Mandatory  yes  or 

no 

12. 

Positive  integer 

or 

zero 

IV. 

INPUT/OUTPUT 

3-6. 

Mandatory  yes  or 

no 

V. 

REFERENCES 

2-6. 

Positive  integer 

or 

zero 

7-9. 

Mandatory  yes  or 

no 

VI. 

DATA 

1-5. 

Positive  integer 

or 

zero 

VII. 

ERROR 

HANDLING 

1. 

Positive  integer 

or 

zero 

2-4. 

Mandatory  yes  or 

no 

VIII. 

EFFICIENCY 

3-5. 

Positive  integer 

or 

zero 

IX. 

PORTABILITY 

1. 

Mandatory  yes  or 

no 

2. 

Positive  integer 

or 

zero 

3-4. 

Mandatory  yes  or 

no 

X. 


FLEXI  BIL  LTY 

1-4.  Mandatory  yes  or  no 


XI.  DYNAMIC  MEASUREMENTS 
1.  Yes,  no,  or  NA 

2-3.  Both  items  must  have  similar  units  of 
measurements  or  zero 

4-7.  Positive  integers  or  zero.  May  be  expressed 
in  units  of  hundreds  or  thousands 
8-9.  Positive  integer  or  zero. 

XII.  INSPECTOR’S  COMMENTS 

Free  format,  limited  to  180  characters. 


APPENDIX  C 

EXPLANATION  OF  METRICS 
(Refs  17,  25) 


explanation:  of  metrics 


Eicn  metric  and  each  metric  element  are  described  in  :ne  following  paragraphs. 
Indication  is  oroviaed  if  the  metric  is  applied  at  the  s/stem  level  or  the 
module  level  and  during  wnich  phases. 

Traceabi 1 i ty 

'R.l  Cross  reference  relating  modules  to  requirements  (design  and  imple¬ 
mentation  phases  at  system  level). 

During  design,  the  identification  of  which  itemized  requirements  are  satis¬ 
fied  in  the  design  of  a  module  are  documented.  A  traceability  matrix  is  an 
example  of  how  this  can  be  done.  During  Implementation,  which  itemized  require¬ 
ments  are  being  satisfied  by  the  module  implementation  are  to  be  identified. 

Some  form  of  automated  notation,  prologue  comments  or  imbedded  comments,  is 
used  to  provide  this  cross'  reference.  The  metric  is  the  Identification  of 
a  tracing  from  requirements  to  design  to  code. 

Completeness 

C? . 1  Completeness  Checklist  (All  three  phases  at  system  level). 

This  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Unambiguous  references  (input,  function,  output). 

Unique  references  to  data  or  functions  avoid  ambiguities  such  as 
a  function  being  called  one  name  by  one  module  and  by  another 
name  by  another  module.  Unique  references  avoid  this  type  of 
ambiguity  in  all  three  phases. 

(2)  All  data  references  defined,  computed,  or  ootained  from  an 
external  source. 

Each  data  element  is  to  nave  a  specific  origin.  At  the 
requirements  level  only  major  global  data  elements  and  a  few 
specific  local  data  elemen*s  may  be  available  tc  be  :hecked 
T  ie  set  of  data  elements  available  for  completeness  checking  a: 
fie  design  level  'ncreases  substantia1 1 y  md  is  to  be  compete 
implementation 


(3)  All  defined  functions  used. 

A  function  which  is  defined  but  not  used  during  a  phase  is 
either  nonfunctional  or  a  reference  to  it  has  been  omitted. 

(4)  All  referenced  functions  defined. 

A  system  is  not  complete  at  any  phase  if  durnny  functions  are 
present  or  if  functions  have  been  referenced  but  not  defined. 

(5)  All  conditions  and  processing  defined  for  each  decision  point. 

Each  decision  point  is  to  have  all  of  its  conditions  and  alter¬ 
native  processing  paths  defined  at  each  phase  of  the  software 
development.  The  level  of  detail  to  which  the  conditions  and  alter 
native  processing  are  described  may  vary  but  the  important  element 
is  that  all  alternatives  are  described. 

(5)  All  defined  and  referenced  calling  seauence  parameters  agree. 

For  each  interaction  between  modules,  the  full  complement  of 
defined  parameters  for  the  interface  is  to  be  jsed.  A  par¬ 
ticular  call  to  a  module  should  not  pass,  for  example,  only  five 
of  the  six  defined  parameters  for  that  module. 

All  problem  reports  resolved. 

At  each  phase  in  the  development,  problem  reports  are  generated. 
Each  is  to  be  closed  or  a  resolution  indicated  to  ensure  s 
comolete  product. 


I 


Consistency 

CS.l  Procedure  Consistency  Measure  (design  and  implementation  at  system 
level ) . 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Standard  Oesign  Representation . 

Flow  charts,  HIPO  charts,  Program  Oesign  Language  -  whichever  form 
of  design  representation  is  used,  standards  for  representing  the 
elements  of  control  flow  are  to  be  established  and  followed.  This 
element  applies  to  design  only.  The  measure  is  based  on  the  number  of 
modules  whose  design  representation  does  not  comply  with  the  standards. 

(2)  Calling  sequence  conventions. 

Interactions  between  modules  are  to  be  standardized.  The  stan¬ 
dards  are  to  be  established  during  design  and  followed  during 
implementation.  The  measure  is  based  on  the  number  of  modules 
which  do  not  comply  with  the  conventions. 

(3)  Input/Output  Conventions . 

Conventions  for  which  modules  will  oerform  I/O,  how  i c  will  be 
accomplished,  and  the  I/O  formats  are  to  be  established  and 
followed.  The  measure  is  based  on  which  modules  do  not  comply  with 
the  conventions. 

(a)  Error  Handling  Conventions. 

A  consistent  method  for  er^or  handling  is  required.  Conven¬ 
tions  established  in  design  are  followed  into  implementation. 

The  measure  is  based  on  the  number  of  modules  which  do  not 
ccmoly  with  the  conventions. 
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CS.2  Data  Consistency  Measure  (Design  and  implementation  at  system  ’evel) 

The  metric  is  the  sum  of  the  scores  of  the  following  aoplicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Standard  data  usage  representation. 

In  concert  with  CS.l  (1),  a  standard  design  representation  for 

oata  usage  is  to  be  established  and  followed.  This  is  a  design  metric 

only,  identifying  the  number  of  modules  which  violate  the  standards. 

(2)  Naming  Conventions . 

Naming  conventions  for  variables  and  modules  are  to  be  established 
and  followed. 

(3)  Consistent  Global  Definitions. 

Global  data  elements  are  to  be  defined  in  the  same  manner  by  all 
modules.  The  measure  is  based  on  the  number  of  modules  in  which 
the  global  data  elements  are  defined  in  an  inconsistent  manner 
for  both  design  and  implementation. 


Accuracy 

AC .  1  Accuracy  Checklist  ( requi rements ,  design,  into lementat ion  phases  at 
system  level),  each  element  is  a  binary  measure  indicating  existence,  or 
absence  of  the  elements.  The  metric  is  the  sum  of  the  scores  of  the 
following  applicable  elements  divided  by  the  number  of  applicable  elements. 

(1)  Error  analysis  performed  and  budgeted  to  module  (reauirements 
phase  only) . 

An  error  analysis  must  be  part  of  the  requirements  analysis  performed 
to  develop  the  requirements  specification.  This  analysis  allocates 
overall  accuracy  requirements  to  the  individual  functions  to  be 
performed  by  the  system.  This  budgeting  of  accuracy  requirements 
provides  definitive  objectives  to  the  module  designers  and 
implementers. 

(2)  A  definitive  statement  of  requi rement  for  accuracy  of  inputs, 
outputs,  processing,  and  constants  (requirements  phase  only). 

See  explanation  above  (1). 

(3)  Sufficiency  of  Math  Library  (design  phase  only). 

The  accuracy  of  the  math  library  routines  utilized  within  the 
system  is  to  be  checked  for  consistency  with  the  overall 
accuracy  objectives. 

(4)  Sufficiency  of  numerical  methods  (design  and  imolementation 
phase)  • 

The  numerical  methods  utilized  within  the  system  are  to  be  consis¬ 
tent  with  the  accuracy  objectives.  They  can  be  checked  at  desion 
and  implementation. 

( z'  Execution  cutouts  within  tole>*irces  i 'molementation  ohase  only 
>*ecui^ing  execution). 

1  final  measure  luring  ieve ’ocnient  testing  •  ;  execution  ;f  'Prv- 
jles  mo  tneck-ng  -'or  iccuracv 


tu tours  . 


£rror  Tolerance 

£T.l  Error  Tolerance  Control  Checklist  (design  and  implementation  phases 
at  system  level). 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements  divided 
by  the  number  of  applicable  elements. 

(1)  Concurrent  processing  centrally  controlled. 

Functions  which  may  be  used  concurrently  are  to  be  controlled 
centrally  to  provide  concurrency  checking,  read/write  locks,  etc. 
Examples  are  a  data  base  manager,  I/O  handling,  error  handl  ng, 
etc.  The  central  control  must  be  considered  at  design  and  then 
implemented. 

(2)  Errors  fixabTe  and  processing  continued- 

When  an  error  is  detected,  the  capability  to  correct  it  on-line 
and  then  continue  processing,  should  be  available.  An  example  is 
an  operator  message  that  the  wrong  tape  is  mounted  and  processing 
will  continue  when  correct  tape  is  mounted.  This  can  be  measured 
at  design  and  implementation. 

(3)  When  an  error  condition  is  detected,  the  condition  is  to  be  Dassed  ud  co 
calling  routine. 

The  decision  of  what  to  do  about  an  error  is  to  be  made  at  a 
level  where  an  affected  module  is  controlled.  This  concept  is 
built  Into  the  design  and  then  implemented. 

ET.2  Recovery  from  Improper  Input  Data  Checklist  (all  three  phases  at 
system  level).  The  metric  is  the  sum  of  the  scores  of  the  following  aopl:- 
cable  elements  divided  by  the  number  of  the  applicable  elements. 
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(1)  A  definitive  statement  of  requirement  for  error  tolerance  of 
input  data. 

The  requirements  specification  must  identify  the  error  tolerance 
capabilities  desired  ( requi rements  phase  only). 

* 2 )  Range  of  values  (reasonableness)  for  items  specified  and  checked 
(design  and  implementation  phases  only). 

The  attributes  of  eacn  input  item  are  to  be  checked  for  reason¬ 
ableness.  Examples  are  checking  items  if  they  must  be  numeric, 
alphabetic,  positive  or  negative,  of  a  certain  length,  nonzero, 
etc.  T,,ese  checks  are  to  be  specified  at  design  and  exist  in 
code  at  implementation. 

(3)  Conflicting  requests  and  illegal  combinations  identified  and  cnecked 
(design  and  implementation  phases  only). 

Checks  to  see  if  redundant  input  data  agrees,  if  combinations  of  param 
eters  are  reasonable,  and  if  requests  are  conflicting  should  be  docu¬ 
mented  In  the  design  and  exist  in  the  code  at  implementation. 

(4)  All  input  is  checked  before  processing  begins  (design  and  imple¬ 
mentation  phases  only). 

Input  checking  is  not  to  stop  at  the  first  error  encountered  bur.  to  co 
tinue  through  all  the  input  and  then  report  all  errors.  Drocess'.  ng  is 
not  to  start  until  the  errors  are  reported  and  either  corrections  are 
made  or  a  continue  processing  command  • s  given. 

(5)  Determination  that  all  data  is  available  prior  to  processing. 

To  avoid  going  through  several  processing  steps  before  incomple''? 
input  data  is  discovered,  checks  for  sufficiency  of  inout  data 
are  to  be  made  orior  to  the  start  of  processing. 

i*  3  Recovery  from  Computational  Failures  Checklist  'i''  tnree  onases  a* 
system  ’evel\  The  metric  is  the  sum  of  the  scores  of  tre  -'oilowmo  aool  - 
:ao le  elements  divided  by  the  number  of  3Dolic3bie  elements. 

\(l 


il)  A  definitive  statement  of  requirement  for  recovery  from  compu¬ 
tational  failures  ( requirements  phase  only). 

The  requirement  for  this  type  error  tolerance  capabi 1 i ty  are  to 
be  stated  during  requirements  phase. 

(2)  Loop  and  multiple  transfer  index  parameters  range  tested  before 
use.  (design  and  implementation  phase  only). 

Range  tests  for  loop  Indices  and  multiple  transfers  are  to  oe 
specified  at  design  and  to  exist  in  code  at  implementation. 

(3)  Subscript  checking  (design  and  implementation  phases  only). 

Checks  for  legal  subscript  values  are  to  be  specified  at  design 
and  coded  during  implementation. 

(A)  Critical  output  parameters  reasonableness  checked  during 
processing  (design  and  implementation  phases  only). 

Certain  range-of- value  checks  are  to  be  made  during  processing  to 
ensure  the  reasonableness  of  final  outputs.  T'nis  is  usually  dene 
only  for  critical  parameters.  These  are  to  be  identified  during 
design  and  coded  during  implementation. 

ET.4  Recovery  from  Hardware  Faults  Checklist  (All  three  Dhases  at  system 
level).  The  metric  is  the  sum  of  scores  from  tne  applicable  elements  di viced 
oy  the  number  of  applicable  elements. 

. !)  A  definitive  statement  of  requirements  for  recovery  f^om  Hardware 
•‘suits  ( requirements  only). 

“he  handl’ng  of  hardware  faults  such  as  arithmetic  faults,  cower 
failure,  clock  internets,  etc.,  are  to  be  specified  luring  require¬ 
ments  phase 


i'  > 


1 2 )  Recovery  from  Hardware  Faults  (design  and  imolementation  phases 
only). 

The  design  specification  and  code  to  provide  the  recovery  from 
the  hardware  faults  identified  in  the  requirements  must  exist 
in  the  design  and  implementation  pnases  respecti vely . 

E7.5  Recovery  from  Device  Errors  Checklist  (all  three  phases  at  system 
level!.  The  metric  is  the  score  given  to  the  applicable  elements  below 
at  each  ohase. 

(1)  A  definitive  statement  of  requirements  for  recovery  from  device 
errors  ( requi rements  only). 

The  handling  of  device  errors  such  as  unexpected  end-of-files 
or  and-of-tape  conditions  or  read/write  failures  are  specified 
during  the  requirements  phase. 

2)  Recovery  from  Oevice  Errors  (design  and  implementation  phases 
only). 

The  design  specification  and  code  to  provide  the  required 

handling  of  device  errors  must  exist  in  the  design  and  implementation 

phases  respecti vely. 


Si  mol icity 

S 1 .  1  Design  Structure  Measure  (design  and  implementation  ohases  at  system 
level).  The  metric  is  the  sum  of  the  scores  of  the  applicable  elements 
divided  by  the  number  of  applicable  elements. 

■'!)  Design  organized  in  -op  sown  fashion. 

A  hierarchy  chart  of  system  -nodules  is  usually  available  or  easy 
to  construct  ffrom  design  documentation.  It  should  -e"'1  ct  t^e 
accepted  notion  of  top  down  design,  'he  system  is  iroan’zeo 
in  a  hi eracrcha  1  tree  structure,  each  ’evel  of  the  t:*oe  ■'eo'-'isents 
’■'wer  levels  of  leta’l  iescn  otions  if  toe  orocessmo 
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U'  Module  independence. 

The  processing  done  within  a  nodule  is  ^ot  to  De  dependent  on  the 
source  of  input  or  the  destination  of  the  output.  This  rule  can 
be  applied  to  the  module  description  during  design  and  the  coded 
module  during  implementation.  The  measure  for  this  element  is 
based  on  the  number  of  modules  which  do  not  comply  with  this  rule. 

(31  Module  processing  not  dependent  on  prior  processing. 

The  processing  done  within  a  module  is  not  to  be  dependent  upon 
knowledge  or  results  of  prior  processing,  e.g.,  the  first  time 
through  the  module,  the  nth  time  through,  etc.  This  rule  is 
applied  as  above  at  design  and  implementation. 

(-1)  Each  module  description  includes  input,  output,  processing, 

1 imi tations . 

Documentation  which  describes  the  input,  output,  processing,  and 
limitations  for  each  module  is  to  be  developed  during  design  and 
available  during  implementation.  The  measure  for  this  element  Is 
based  on  the  number  of  modules  which  do  not  have  this  information 
documented. 

(5)  Each  module  has  single  entrance,  single  exit. 

Determination  of  the  number  of  modules  that  violate  this  rule  at 
design  and  implementation  can  be  made  and  is  the  basis  for  the  metr 

; 5  Size  of  data  base . 

The  size  of  the  data  base  in  terms  of  the  number  of  unique  data 
items  contained  in  the  data  base  relates  to  the  design  structure 
of  the  software  system.  A  data  item  is  a  unique  data  element 
for  example  an  individual  data  entry  or  data  e Id . 
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17)  Compartamental i zation  of  Data  Base 

The  structure  of  the  data  base  also  is  represented  oy  its 
modularization  or  how  it  is  decomposed.  The  size  determined 
In  (6)  above  divided  by  the  number  of  data  sets  provided  this 
measure.  A  data  set  corresponds  to  the  first  level  of  decom¬ 
position  of  a  data  base,  a.g.,  a  set  in  a  CCOASYL  data  base, 
a  record  in  a  file  system,  a  COMMON  in  FORTRAN,  or  a  Data 
Block  in  a  COMPOOl  system 

SI.  3  Oata  and  Control  flow  Complexity  measure  (Design  and  implementation 
phases ) . 

This  metric  can  be  measured  from  the  design  representation  (e.g.,  flowcharts) 
and  the  code  automatically.  Path  flow  analysis  and  variable  set/use  informa¬ 
tion  along  each  path  is  utilized.  A  variable  is  considered  to  be  'live'  at  a 
node  if  it  can  be  used  again  along  that  path  in  the  program.  The  com¬ 
plexity  measure  is  based  on  summing  the  'liveness’  of  all  variables  along 
all  paths  in  the  program.  It  is  normalized  by  dividing  it  by  the  maximum 
complexity  of  the  program  (all  variables  live  along  all  paths). 


SI.  4  Measure  of  Simplicity  of  Coding  Techniques  (Implementation  phase 
applied  at  module  level  first).  The  metric  at  the  system  level  is  an 
averaged  quantity  of  all  the  module  measures  for  the  system.  The  module 
measure  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Module  flow  top  to  bottom. 

This  is  a  binary  measure  of  the  logic  flow  of  a  module.  If  it 
flows  too  to  bottom,  it  is  given  a  value  of  1,  if  not  a  0. 

(2)  Negative  Boolean  or  complicated  Compound  Boolean  expressions 
used. 

Comoound  expressions  involving  two  or  more  Boolean  operators  and 
negation  can  often  be  avoided.  These  types  of  expressions  add 
to  the  complexity  or  the  module.  The  measure  is  based  on  the 
numoer  of  these  complicated  expressions  oer  executable  statement 
• n  the  module. 


Hi 


(3)  Jumps  in  and  out  of  loops. 

Loops  within  a  (nodule  should  have  one  entrance  and  one  exit. 

This  measure  is  based  on  the  number  of  loops  which  comply  with  this 
rule  divided  by  the  total  number  of  loops. 

(4)  Loop  index  modified. 

■Modification  of  a  loop  Index  not  only  complicates  the  logic  of  a 
module  but  causes  severe  problems  while  debugging.  This  measure 
is  based  on  the  number  of  loop  indices  which  are  modified  divided 
by  the  total  number  of  loops. 

(5)  Module  Is  not  sel f-modifylng  . 

If  a  module  has  the  capability  to  modify  its  processing  logic  it  becomes 
very  difficult  to  recognize  what  state  it  Is  in  when  an  error  occurs.  In 
addition,  static  analysis  of  the  logic  is  more  difficult.  This  measure 
emphasizes  the  added  complexity  of  self-modifying  modules. 

(6)  Number  of  statement  labels. 

This  measure  is  based  on  the  premise  that  as  more  statement  labels 
are  added  to  a  module  the  more  complex  it  becomes  to  understand. 

(7)  Nesting  level. 

The  greater  the  nesting  level  of  decisions  or  loops  within  a  mod¬ 
ule,  the  greater  the  complexity.  The  measure  is  the  inverse  of 
the  maximum  nesting  level. 

;3)  Number  of  branches. 

The  more  paths  or  branches  that  are  present  in  a  module,  the 
greater  the  complexity.  This  measure  is  based  on  -he  number 
of  decision  statements  oer  executable  statements. 


17? 


t  ^ 


(9)  Number  or  SOTO’s. 

Much  has  been  written  in  the  literature  about  the  /irtues  of 
avoiding  GOTO's.  This  measure  is  based  on  the  number  of  GOTO 
statements  per  executable  statement. 

(10)  Variable  mix  in  a  module. 

From  a  somplicity  viewpoint,  local  variables  are  far  better  than 
global  variables.  This  measure  is  the  ratio  of  internal  (local) 
variables  to  total  (internal  (local)  plus  external  (global)) 
varialbes  within  a  module. 

(11 )  Variable  density. 

The  more  used  of  variables  in  a  module  the  greater  the  complexity 
of  that  module.  This  measure  is  based  on  the  number  of  variable 
uses  In  a  module  divided  by  the  maximum  possible  uses. 

Modularity 

MO. 2  Modular  Implementation  Measure  (design  and  implementation  phases  at 
system  level).  The  metric  is  the  sum  of  the  scores  of  the  following  ap¬ 
plicable  elements  divided  by  the  number  of  applicable  elements. 

(1)  Hierarchical  Structure. 

The  measure  refers  to  the  modular  implementation  of  the  top  down 
design  structure  mentioned  in  SI.l  (1).  The  hierarchical  struc¬ 
ture  obtained  should  exemplify  the  following  rules:  Interactions 
between  modules  are  restricted  to  flow  of  control  between  a  pre¬ 
decessor  module  and  its  Immediate  successor  modules.  This  mea¬ 
sure  is  based  on  the  number  of  violations  to  this  rule. 

i 2)  Module  Size  Profile. 

The  standard  module  size  of  procedural  statements  can  vary.  100 
statements  has  been  mentioned  in  the  literature  frequently. 

This  ^measure  is  based  on  the  number  of  orocedural  statements  :n 
a  module. 
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(3)  Controlling  parameters  defined  by  calling  module. 

The  next  four  elements  further  elaborate  on  the  control  and 
interaction  between  modules  referred  to  by  (1)  above.  The 
calling  module  defines  the  controlling  parameters ,  any  input 
data  required,  and  the  output  data  required.  Control  must 
also  be  returned  to  the  calling  module.  This  measure  is  based 
on  the  number  of  calling  parameters  which  are  control  para¬ 
meters.  The  next  three  are  based  on  whether  a  rule  is  vio¬ 
lated.  They  can  be  measured  at  design  and  implementation. 

(4)  Input  data  controlled  by  calling  module. 

See  (3)  above. 

(5)  Output  data  provided  to  calling  module. 

See  (3)  above. 

(6)  Control  returned  to  calling  module. 

See  (3)  above. 

(7)  Modules  do  not  share  temporary  storage. 

This  is  a  binary  measure,  1  if  modules  do  not  share  temporary 
storage  and  0  if  they  do.  It  emphasizes  the  loss  of  module 
independence  if  temporary  storage  is  shared  between  modules. 

Generali ty 

GE.i  Extent  to  which  modules  are  referenced  by  other  modules  '.design  and 
implementation  at  system  level).  This  metric  provides  a  measure  of  the 
generality  of  the  modules  as  they  are  used  in  the  current  system,  -i  nod¬ 
ule  is  considered  to  be  more  general  in  nature  if  it  is  used  (referenced) 
by  more  than  one  module.  The  number  of  these  common  modules  divided  by 
the  total  number  of  modules  provides  the  measure. 
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of  all  possible  inputs  to  which  a  function  can  oe  applied  the 
less  general  it  is.  Thus,  this  measure  is  based  on  the  number 
of  modules  which  are  data  value  limited.  This  can  be  deter¬ 
mined  at  design  and  implementation. 

Expandabi 1 i ty 

EX.l  Data  Storage  Expansion  Measure  (design  and  implementation  phase  at 
system  level).  The  metric  is  the  sum  of  the  scores  of  the  following  appli¬ 
cable  elements  divided  by  the  number  of  applicable  elements. 

(1)  Logical  processing  Independent  of  storage  specification/ require¬ 
ments.  The  logical  processing  of  a  module  is  to  be  independent 
of  storage  size,  buffer  space,  or  array  sizes.  The  design  pro¬ 
vides  for  variable  dimensions  and  dynamic  array  sizes  to  be  defined 
parametrically.  The  metric  is  based  on  the  number  of  modules  con¬ 
taining  hard-coded  dimensions  which  do  not  exemplify  this  concept. 

(2)  Percent  of  memory  capacity  uncommitted  (implementation  only). 

The  amount  of  memory  available  for  expansion  is  an  important  mea¬ 
sure.  This  measure  identifies  the  percent  of  available  memory 
which  has  not  been  utilized  in  implementing  the  current  system. 

EX. 2  Extensibility  Measure  (design  and  implementation  phases  at  the  system 
level).  The  metric  is  the  sum  of  the  scores  of  the  following  apoli cable 
elements  divided  by  the  number  of  applicable  elements. 
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S £.2  Implementation  for  Generality  Measure  (design  and  implementation 
phases).  This  metric  is  the  sum  of  the  scores  of  the  following  applicable 
elements  divided  by  the  number  of  applicable  elements. 

(1)  Input,  processing,  output  functions  are  not  mixed  in  a  single 
function. 

A  module  which  performs  I/O  as  well  as  processing  is  not  as 
general  as  a  module  which  simply  accomplishes  the  processing. 

This  measure  is  based  on  the  number  of  modules  that  violate 
this  concept  at  design  and  implementation. 

(2)  Application  and  machine  dependent  functions  are  not  mixed  in 
a  single  module  (Implementation  only). 

Any  references  to  machine  dependent  functions  within  a  module 
lessens  its  generality.  An  example  would  be  referencing  the 
system  clock  for  timing  purposes.  This  measure  is  based  on  the 
number  of  machine  dependent  functions  in  a  module. 


(3)  Processing  not  data  volume  limited. 

A  module  which  has  been  designed  and  coded  to  accept  no  more 
than  i 00  data  item  inputs  for  processing  is  certainly  not  as 
general  in  nature  as  a  module  whicn  will  accept  any  volume  of 
input.  This  measure  is  based  on  the  number  of  modules  which 
are  designed  or  implemented  to  be  data  volume  limited. 

(4)  Processing  not  data  value  limited. 

4  previously  identified  element,  ST. 2  (2)  of  -rror  Tolerance 
dealt  with  cfecking  input  for  reasonableness  •  This  capad-i 1 ;  ty 
is  required  to  prevent  providing  data  to  a  function  for  wrnch 
it  is  not  defined  or  its  degree  of  precision  is  not  acceptable, 
etc.  This  is  necessary  capability  from  jn  error  tolerance 
/iewpoint.  r>om  a  generally  /’ewpoint,  tne  smaller  the  s..oset 
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(1)  Accuracy,  convergence ,  timing  attributes  wnich  control  processing 
are  parametric. 

A  module  whicn  can  provide  varying  degrees  of  <_  nvergence  or  timing 
to  achieve  greater  precision  provides  this  attribute  of  extensibil¬ 
ity.  Hard-coded  control  parameters,  counters,  clock  values,  etc. 
violate  this  measure.  This  measure  is  based  on  the  number  of  mod¬ 
ules  which  do  not  exemplify  this  characteristic.  A  determination 
can  be  made  during  design  and  implementation. 

(2)  Modules  table  driven. 

The  use  of  tables  within  a  module  facilitates  different  representa¬ 
tions  and  processing  characteristics.  This  measure  which  can  be 
applied  during  design  and  implementation  is  based  on  the  numoer  of 
modules  which  are  not  table  driven. 

(3)  Percent  of  speed  capacity  uncommitted  (implementation  only). 

A  certain  function  may  be  required  in  the  performance  requirements 
specification  to  be  accomplished  in  a  specified  time  for  overall 
timing  objectives.  The  amount  of  time  not  used  by  the  current 
implementation  of  the  function  is  processing  time  available  -'or 
potential  expansion  of  computational  capabilities.  This  measure 
identifies  the  percent  of  total  processing  time  that  is 
unconwi tted. 

Instrumentation 

IN .  1  'Module  testing  measure  (design  and  implementation  phases,  first  at  mod¬ 
ule  level  then  system  level).  The  system  level  metric  is  an  average  of  all 
module  measures .  The  module  measure  is  the  average  score  of  the  following 
two  elements: 

(I)  Path  coverage- 

Plans  for  testing  the  various  paths  within  a  module  shoul  1  be  made 
during  design  and  the  test  cases  actually  develooed  dunnj  imple¬ 
mentation.  This  measure  identifies  the  number  of  oaths  panned  :o 
be  tested  iivided  oy  the  total  number  of  oaths. 

2]  Inout  parameters  boundary  ‘est?d. 

“he  othe*'  jsoect  of  nodule  tasting  nvolves  testing  the  incut 
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ranges  to  the  module,  “his  is  lone  by  exercising  the  module  at  the 
various  boundary  values  of  the  input  parameters.  Plans  to  do  this 
must  be  specified  during  design  and  coded  during  impl ementation. 

The  measure  is  the  number  of  parameters  to  be  boundary  tested 
divided  by  the  total  number  of  parameters. 

IN. 2  Integration  Testing  Measure  (design  and  implementation  phases  at  system 
level).  The  metric  is  the  averaged  score  of  the  following  two  elements. 

(1)  Module  interfaces  tested. 

One  aspect  of  integration  testing  is  the  testing  of  all  module  to 
module  Interfaces.  Plans  to  accomplish  this  testing  are  prepared 
during  design  and  the  tests  are  developed  during  implementation. 

The  measure  is  based  on  the  number  of  interfaces  to  be  tested 
divided  by  the  total  number  of  interfaces. 

(2)  Performance  requirements  (timing  and  storage)  coverage. 

The  second  aspect  of  integration  testing  involves  checking  for  com¬ 
pliance  at  the  module  and  subsystem  level  with  the  performance 
requirements.  This  testing  is  planned  during  design  and  the  tests 
are  developed  during  implementation.  The  measure  is  the  numoer 
of  performance  requirements  to  be  tested  divided  by  the  total 
number  of  performance  requirements . 


IN. 3  System  Testing  Measure  (design  and  implementation  oh.  ses  at  the  system 
level).  The  metric  is  the  averaged  score  of  the  two  elements  below. 

(1)  Module  Coverage. 

One  aspect  of  system  testing  which  can  be  measured  as  early  as  the 
design  phase  is  the  equivalent  to  path  coverage  at  the  module  level. 
For  all  system  test  scenarios  planned,  the  percent  of  all  of  the 
modules  to  be  exercised  is  important. 

(2)  Identification  of  test  inputs  and  outputs  in  sumary  form. 

The  results  of  tests  and  the  manner  in  which  these  results  are 
displayed  are  very  important  to  the  effectiveness  of  testing.  This 
is  especially  fue  during  system  testing  because  of  the  potentially 
large  volume  of  input  and  output  data.  This  measure  simply  identi¬ 
fies  if  the  capability  exists  to  display  test  inouts  and  outputs 
in  a  summary  fashion.  The  measure  can  be  applied  to  the  plans 
and  specifications  in  the  design  phase  and  the  development  of 
this  capability  during  implementation. 

Self  Oescriotiveness 

SO. I  Quantity  of  Comments  (implementation  phase  at  module  level  first  and 
then  system  level).  The  metric  is  the  number  of  comment  lines  divided  by  the 
total  number  of  lines  in  each  module.  Blank  lines  are  not  counted.  The 
average  /alue  is  computed  for  the  system  level  metric. 
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SO. 2  Effectiveness  of  Comments  Measure  (implementation  phase  at  system  level) 
The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 


(1)  Modules  have  standard  formatted  prologue  comments. 

This  information  is  extremely  valuable  to  new 

personnel  who  have  to  work  witn  the  software  after  development, 
performing  maintenance,  testing,  changes,  etc.  The  measure  at 
the  system  level  is  based  on  the  number  of  modules  which  do  not 
comply  with  a  standard  format  or  do  not  provide  complete  information. 

(2)  Comments  set  off  from  code  in  uniform  manner. 

Blank  lines,  bordering  asterisks,  specific  card  columns  are  some  of 
the  techniques  utilized  to  aid  in  the  identification  of  comments. 

The  measure  is  based  on  the  number  of  modules  which  do  not  follow 
the  conventions  established  for  setting  off  the  comments. 

(3)  All  transfers  of  control  and  destinations  commented. 

This  form  of  comment  aids  in  the  understanding  and  ability  to  follow 
the  logic  of  the  module.  The  measure  is  based  on  the  number  of 
modules  which  do  not  comply. 

(4)  All  machine  dependent  code  commented. 

Comments  associated  with  machine  dependent  code  are  important  not 
only  to  explain  what  is  being  done  but  also  serves  to  identify 
that  portion  of  the  module  as  machine  dependent.  The  metric  is 
based  on  the  number  of  modules  wnich  do  not  nave  the  machine 
dependent  code  commented. 

(5'1  All  non-standard  HOI  statements  coronented. 

A  similar  explanation  to  4)  above  is  aool -cable  here. 


(6)  Attributes  of  all  declared  variables  commented. 

The  usage,  properties,  units,  etc.,  of  variables  are  to  be  explained 
in  contnents.  The  measure  is  based  on  the  number  of  modules  wnicn  do 
not  follow  bhls  practice. 

(7)  Comments  do  not  just  repeat  operation  described  in  language. 

Comments  are  to  describe  why  not  what.  A  comment,  increment  A  by  1, 
for  the  statement  A»A+1  provides  no  new  information.  A  comment, 
increment  the  table  look-up  index,  is  more  valuable  for  under¬ 
standing  the  logic  of  the  module.  The  measure  is  based  on  the 
number  of  modules  in  which  conments  do  not  explain  the  why's. 

SO. 3  Oescriptiveness  of  Implementation  Language  Measure  (implementation 
phase  at  system  level).  The  metric  is  the  sum  of  the  scores  of  the  following 
applicable  elements  divided  by  the  number  of  applicable  elements. 

(1)  High  Order  Language  used. 

An  HOL  is  much  more  sel f-descripti ve  than  assembly  language.  The 
measure  is  based  on  the  number  of  modules  which  are  implemented, 
in  whole  or  part,  in  assembly  or  machine  language. 

(2)  Variable  names  (mnemonics)  descriptive  of  physical  or  functional 
property  represented. 

While  the  metric  appears  very  subjective,  it  is  suite  easy  :o 
identify  if  variaole  names  have  been  chosen  with  sel‘- 
descri pti veness  in  mind.  T’nr«>e  variaole  names  such  is  NAME  . 

POSIT,  SALPY  are  far  better  and  more  easily  ••ecogniced  as  pes¬ 
ter  than  A1 ,  A2 ,  A3.  The  measure  is  based  on  she  number  of 
modules  which  do  not  utilize  descriptive  names. 


(3)  Source  code  logically  blocked  and  indented. 

Tecnniques  such  as  blocking,  paragraphing,  indenting  for  specifi: 
constructs  are  well  established  and  are  to  be  followed  uniformly 
with  a  system.  This  measure  is  based  on  the  number  of  modules 
which  do  not  comply  with  a  uniform  technique. 

(4)  One  statement  per  line. 

The  use  of  :ontinuation  statements  and  multiple  statements  oer  line 
causes  difficulty  in  reading  the  code.  The  measure  is  the  number 
of  continuations  plus  the  number  of  multiple  statement  lines  divioed 
by  the  total  number  of  lines  for  each  module  and  then  averaged  over 
all  of  the  modules  in  the  system. 


Execution  Efficiency 

EE.l  Performance  Requirements  allocated  to  design  (design  phase  at  system 
level).  Performance  requirements  for  the  system  must  be  broken  down  and 
allocated  appropriately  to  the  modules  during  the  design.  This  metric  simply 
Identifies  If  the  performance  requirements  have  (1)  or  have  not  (0)  been 
allocated  during  the  design. 


EE. 2  Iterative  Processing  Efficiency  Measure  (design  and  implementation 
ohases  at  module  level  first).  The  metric  at  the  module  level  is  the  sum  of 
the  scores  of  the  following  applicable  elements  divided  by  the  number  of 
elements.  At  the  system  level  it  is  an  averaged  score  for  all  of  the  modules. 


(1)  Non-loop  dependent  computations  keDt  out  of  looo. 

Such  practices  as  evaluating  constants  in  a  loop  are  *o  Se  avoided. 
This  measure  is  based  on  the  number  of  non-looD  dependent  statements 
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found  in  all  loops  in  a  module.  This  is  to  be  measured  from  a 
detailed  design  representation  during  design  and  from  the  code 
during  implementation. 

(2)  Performance  Optimizing  Compiler/Assembly  language  used  (implementation 
only) . 

This  is  a  binary  measure  which  identifies  if  a  performance  ootimizing 
compiler  was  used  (1)  or  if  assembly  language  was  used  to  accomplish 
performance  optimization  (l)  or  not  (0). 

(3)  Compound  expressions  defined  once  (implementation  only). 

Repeated  compound  expressions  are  to  be  avoided  from  an  efficiency 
standpoint.  This  metric  is  based  on  the  number  of  compound 
expressions  which  appear  more  than  once. 

(4)  Number  of  overlays. 

The  use  of  overlays  requires  overnead  as  far  as  processing  time. 

This  measure,  the  inverse  of  the  number  of  overlays,  reflects  that 
overhead.  It  can  be  applied  during  design  when  the  overlay  scheme 
is  defined  and  during  implementation. 

(5)  Free  of  bit/byte  packing/unpacking  in  loops. 

This  is  a  binary  measure  indicating  the  overhead  Involved  in  bit/byte 
packing  and  unpacking.  Placing  these  activities  within  looos  should 
be  avoided  if  oossible. 


(.51  Module  linxages  (implementation  only,  requires  execution). 

This  measure  essentially  represents  the  inter-module  communication 
overhead.  The  measure  is  based  on  the  amount  of  execution  time 
spent  during  module  to  module  communication. 

(?)  Operating  System  linkages  (implementation  only,  requires  execution). 
This  measure  represents  the  module  to  OS  communication  overhead. 

The  measure  is  based  on  the  amount  of  execution  time  spent  during 
module  to  OS  communications . 

(3)  Efficient  Use  of  storage  facility. 

This  measure  represents  an  evaluation  of  the  utility  of  the  storage 
faci 1 i ty . 

EE. 3  Oata  Usage  Efficiency  Measure  (design  and  impleme.itation  phases  applied 
at  module  level  first).  The  metric  at  the  module  level  is  the  sum  of  the 
scores  of  the  following  applicable  elements  divided  by  the  number  of  applicable 
elements.  The  system  metric  is  the  averaged  value  of  all  of  the  module  metric 
va I ues . 

(1)  Oata  grouped  for  efficient  processing. 

The  data  utilized  by  any  module  is  to  be  organized  n  the  data  base, 
buffers,  arrays,  etc.,  in  a  manner  which  facilitates  efficient 
processing.  The  data  organization  during  design  and  implementation  is 
to  be  examined  to  provide  this  binary  measure. 

[2]  Variables  initialized  when  declared  (implementation  only'. 

This  measure  is  based  on  the  number  of  /a tables  used  in  a  module 
which  are  not  initialized  when  declared. 


Efficiency  is  lost  when  variab  es  are  initialized  during  execution 
of  a  function  or  repeatedly  initialized  during  iterative  processing. 


(3)  No  mix-mode  expressions  (implementation  only). 

Processing  overhead  is  consumed  by  mix-mode  expressions  which  are 
otherwise  unnecessary.  This  measure  is  based  on  the  number  of  mix¬ 
mode  expressions  found  in  a  module. 

(4)  Common  choice  of  units/types. 

For  similar  reasons  as  expressed  above  (3)  this  convention  is  to  be 
followed.  The  measure  is  the  inverse  of  the  number  of  operations 
performed  which  have  uncommon  units  or  data  types. 

(5)  Data  indexed  or  referenced  for  efficient  processing. 

Not  only  the  data  organization,  (1)  above,  but  the  linkage  scheme 
between  data  items  effects  the  processing  efficiently.  This  is  a 
binary  measure  of  whether  the  indexing  utilized  for  the  data  was 
chosen  to  facilitate  processing. 

Storage  Efficiency 

SE.l  Storage  Efficiency  Measure  (design  and  implementation  phases  at  module 
level  first  then  system  level).  The  metric  at  the  module  level  is  the  sum  of 
the  scores  of  the  following  aoplicable  elements  divided  by  the  number  of 
applicable  elements.  The  metric  at  the  system  level  is  the  avenged  value  of 
all  of  the  module  metric  values. 

(1)  Storage  Requirements  allocated  to  design  (design  phase  only). 

The  storage  requirements  for  the  system  are  to  be  allocated  to  the 
individual  modules  during  Jesign.  This  measure  is  a  b'narv  measure 
of  whether  that  allocation  s  explicitly  made  1  or  not  '1 : . 


(2)  Virtual  Storage  Facilities  Used. 

The  use  of  virtual  storage  or  paging  techniques  enhances  the 
storage  efficiency  of  a  system.  This  is  a  binary  measure  of  whether 
these  techniques  are  planned  for  and  used  (l)  or  not  (0). 

(3)  Coronon  data  defined  only  once  (implementation  only). 

Often,  global  data  or  data  used  commonly  are  defined  more  than 
once.  This  consumes  storage.  This  measure  is  based  on  the  number 
of  variables  that  are  defined  in  a  module  that  have  been  defined 
elsewhere. 

(4)  Program  Segmentation. 

Efficient  segmentation  schemes  minimize  the  maximum  segment  length 
to  minimize  the  storage  requirement.  This  measure  is  based  on 
the  maximum  segment  length.  It  is  to  be  applied  during  design  when 
estimates  are  available  and  during  implementation. 

(5)  Dynamic  memory  management  used. 

This  is  a  binary  measure  emphasizing  the  advantages  of  using  dy¬ 
namic  memory  management  techniques  to  minimize  the  amount  of 
storage  required  during  execution.  This  is  planned  during  design 
and  used  during  implementation. 

(5)  Oata  packing  used  (implementation  only). 

While  data  packing  was  discouraged  in  EE. 2  (5)  in  loops  because  of 
the  overhead  it  adds  to  processing  time,  in  general  it  is  bene¬ 
ficial  from  a  storage  efficiency  viewpoint.  This  binary  measure 
applied  during  implementation  recognizes  this  ‘'act. 


(7)  Storage  optimizing  compi ler/assemoly  language  used  ( imp lementa cion 
only). 

This  binary  measure  is  similar  to  EE. 2  (2)  except  from  the  view¬ 
point  of  storage  optimization. 

Access  Control 

AC. 1  Access  Control  Checklist  (all  three  phases  at  system  level). 

The  metric  is  tne  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  User  I/O  Access  controls  provided. 

Requirements  for  user  access  control  must  be  identified  during  the 
requirements  phase.  Provisions  for  identification  and  password 
checking  must  be  designed  and  implemented  to  comply  with  the  require¬ 
ments.  This  binary  measure  applied  at  all  three  phases  identifies 
whether  attention  has  been  placed  on  this  area. 

(2)  Oata  8ase  Access  controls  provided. 

This  binary  measure  Identifies  whether  requirements  for  data  base 
controls  nave  oeen  specified,  designed  and  the  capabilities  impie- 
mentated.  Examples  of  data  base  access  controls  are  authorization 
tables  and  privacy  locks. 


(3)  Memory  protection  across  tasks. 

Similar  to  (1)  and  (2)  above,  this  measure  identifies  the  progression 
from  a  requirements  statement  to  implementation  of  memory  protection 
across  tasks.  Examples  of  this  type  of  protection,  often  times  pro¬ 
vided  to  some  degree  by  the  operating  system,  are  preventing  tasks  from 
invoking  other  tasks,  tasks  from  accessing  system  registers,  and  the 
use  of  privileged  comnands. 

Access  Audit 

AA.l  Access  Audit  Checklist  (all  three  phases  at  system  level). 

The  metric  is  the  averaged  score  of  the  following  two  elements. 

(1)  Provisions  for  recording  and  reporting  access. 

A  statement  of  the  requirement  for  this  type  capao.iity  must  exist  in 
the  requirements  specification.  It  is  to  be  considered  in  the  design 
specification,  and  coded  during  Implementation.  This  binary  metric 
applied  at  all  three  phases  identifies  whether  these  steps  are 
being  taken.  Examples  of  the  provisions  which  might  be  considered 
would  be  the  recording  of  terminal  linkages,  data  file  accesses, 
and  jobs  run  by  user  identification  and  time. 

(2)  Provisions  for  immediate  indication  of  access  violation. 

In  addition  to  (1)  above,  access  audit  capabilities  required 
might  include  not  only  recording  accesses  but  immediate  identifica¬ 
tion  of  unauthorized  accesses,  whether  intentional  or  net.  T'iis 
measure  traces  the  requirement,  design,  and  implementation  o* 
provisions  for  this  capability. 


Qpcraoll itv 

OP. I  Operability  Checklist  (all  three  phases  at  system  level). 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  All  steps  of  operation  described. 

This  binary  measure  applied  at  all  three  phases  identifies  whether 
the  operating  characteristics  have  been  described  in  the  require¬ 
ments  specification,  and  if  this  description  has  been  transferred 
into  an  implementable  description  of  the  operation  (usually  in  an 
operator's  manual).  The  description  of  the  operation  should  cover 
the  normal  sequential  steps  and  all  alternative  steps. 

(2)  All  error  conditions  and  responses  appropriately  described  to 
operator. 

The  requirement  for  this  capability  must  appear  in  the  requirements 
specification,  must  be  considered  durinq  design,  and  coded  during 
implementation.  Error  conditions  must  be  clearly  identified  by 
the  system.  Legal  responses  for  all  conditions  are  to  be  either 
documented  and/or  prompted  by  the  system.  This  is  a  binary  mea¬ 
sure  to  trace  the  evolution  and  implementation  of  these  capabilities 

(3)  Provisions  for  operator  to  interrupt,  obtain  status,  save,  modify, 
and  continue  processing. 

The  capabilities  provided  to  the  operator  must  be  considered  during 
the  requirements  phase  and  then  designed  and  imDlementad .  examples 
of  operator  capabilities  include  halt/resume  and  check  oointing. 

This  is  a  binary  measure  to  trace  the  evolution  of  these 
capabilities. 

(A)  Number  of  operator  actions  reasonable  (implementation  only,  re¬ 
quires  execution). 

Toe  number  of  operator  errors  can  be  rel  ted  d’ recti y  to  toe  oumbe- 
of  actions  reouired  during  a  “me  oericd.  This  measure  is  cased 
the  amount  of  time  spent  '•edu’-’ng  manual  bDerator  action?  n video 
by  the  *o ta  1  time  '•equi^ed  ‘  ar  r ;ot>. 


(5)  Job  set  up  and  tear  down  procedures  described  (implementation  only). 
The  specific  tasks  involved  in  setting  up  a  job  and  completing  it 
are  to  be  described.  This  is  usually  documented  duriny  tne  imple¬ 
mentation  phase  when  the  final  version  of  the  system  is  fixed. 

This  is  a  binary  measure  of  the  existence  of  that  description. 

(6)  Hard  copy  log  of  interactions  maintained  (design  and  implementation 
phases) . 

This  is  a  capability  that  must  be  planned  for  in  design  and  coded 
during  Implementation.  It  assists  in  correcting  operational  errors, 
improving  efficiency  of  operation,  etc.  This  measure  identifies 
whether  it  is  considered  in  the  design  and  implementation  phases  (1) 
or  not  (0). 

(7)  Operator  messages  consistent  and  responses  standard  (design  and 
implementation  phases). 

This  is  a  binary  measure  applied  during  design  and  implementation  to 
insure  that  the  interactions  between  the  ooerator  and  the  system  are 
simple  and  consistent.  Operator  responses  such  as  YES,  NO,  GO,  STOP, 
ere  concise,  simple,  and  can  be  consistently  used  throughout  a  system. 
Lengthy,  differently  formated  responses  not  only  provide  difficult'/ 
to  the  operator  but  also  require  complex  error  checking  routines. 

Training 

TN.  1  Training  Checklist  (design  and  implementation  at  system  level).  The 
metric  is  the  sum  of  the  scores  of  the  following  applicable  elements  divided  by 
the  number  of  applicable  elements. 

(1)  Lesson  PI ans/Training  Material  develooed  for  operators,  end  users, 
maintainers  (implementation  ohase  only). 

This  is  a  binary  measure  of  whether  this  tyoe  documentation  ' s 
orovided  during  the  imolementation  phase. 
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(2)  Realistic  simulated  exercises  provided  (implementation  only:. 

This  is  a  binary  measure  of  whether  exercises  which  represent  the 
operational  environment,  are  developed  during  the  implementation 
phase  for  use  in  training. 

(3)  Sufficient  'help'  and  diagnostic  information  available  on-line. 

This  is  a  binary  measure  of  whether  the  capability  to  aid  the 
operator  in  famil iarizatlon  with  the  system  has  been  designed  and 
built  into  the  system.  Provision  of  a  list  of  legal  commands  or  a 
list  of  the  sequential  steps  involved  in  a  process  are  examples. 

Communicativeness 

CM.l  User  Input  Interface  Measure  (all  three  phases  at  system  level). 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements  divi¬ 
ded  by  the  number  of  applicable  elements. 

(1)  Oefault  values  defined  (design  and  implementation). 

A  method  of  minimizing  the  amount  of  input  required  is  to  provide 
defaults.  This  measure,  applied  during  design  and  implementation, 
is  based  on  the  number  of  defaults  allowed  divided  by  the  total 
number  of  input  parameters. 

(2)  Input  formats  uniform  (design  and  implementation). 

The  greater  the  number  of  input  formats  there  are  the  more  difficult 
the  system  is  to  use.  This  measure  is  bated  on  the  total  number  of 
inout  formats. 

(3)  Each  input  record  sel f- identifying. 

Input  records  wnich  have  sel f- identifying  codes  enhance  the  uccuracv 
of  user  inputs.  This  measure  is  based  on  the  number  of  inout 
records  that  are  not  self  identifying  divided  bv  the  total  -umber 
input  records.  It  •;  to  be  aool'ed  3"  ies’3n  ind  ’mo i ementa : ■ on . 


1,4)  Input  can  be  verified  by  user  prior  to  execution  (design  and 
implementation) . 

The  capability,  displaying  input  upon  request  or  ecnoing  the  input 
automatically,  enables  the  user  to  check  his  inputs  before 
processing.  This  Is  a  measure  of  the  existence  of  the  design  and 
implementation  of  this  capability. 

(5)  Input  terminated  by  explicitly  defined  logical  end  of  input  (design 
and  implementation). 

The  user  should  not  have  to  provide  a  count  of  Input  cards.  This  is 
a  binary  measure  of  the  design  and  implementation  of  this  capability. 

(5)  Provision  for  specifying  input  from  different  media. 

The  flexibility  of  input  must  be  decided  during  the  requirements 
analysis  phase  and  followed  through  during  design  and  implementation. 
This  is  a  binary  measure  of  the  existence  of  the  consideration 
of  this  capability  during  all  three  phases. 

C.M.2  User  Output  Interface  Measure  (all  three  ohases  at  system  level). 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements  divided 
by  the  number  of  applicable  elements. 

(1)  Selective  Output  Controls. 

The  existence  of  a  requirement  for,  design  for,  and  implementation 
of  selective  output  controls  is  indicated  by  this  binary  measure. 
Selective  controls  include  choosing  specific  outputs,  output  formats, 
amount  of  output,  etc. 

(2)  Outputs  have  unique  descriptive  user  oriented  labels  design  and 
implementation  only). 

This  is  a  binary  measure  of  the  design  and  implementation  o*  unique 
output  laoels.  In  addition,  trie  labe's  are  to  be  descriptive  to  toe 
user,  this  includes  not  on  i  y  the  labels  *nicn  ire  used  to 
an  output  -eoort  out  also  toe  "itle.  lo’umn  leadings.  jtc.  ■«ithm  that 
report . 
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(3)  Outputs  have  user  oriented  units  (design  and  implementation;. 

This  is  a  binary  measure  which  extends  (2'  above  to  the  individual 
output  items. 

(4)  Uniform  output  labels  (design  and  implementation). 

This  measure  corresponds  to  CM.l  (2)  above  and  is  the  inverse  of 
the  number  of  different  output  formats. 

(5)  Logical  groups  of  output  separated  for  user  examination  (design 
and  implementation). 

Utilization  of  top  of  page,  blank  lines,  lines  of  asterisks,  etc., 
provide  for  easy  identification  of  logically  grouped  output.  This 
binary  measure  identifies  if  these  techniques  are  used  during  design 
and  implementation. 

(o)  Relationship  between  error  messages  and  outputs  is  unambiguous 
(design  and  implementation). 

This  is  a  binary  measure  applied  during  design  and  implementation 
which  Identifies  if  error  messages  will  be  directly  related  to  the 
output. 

(7)  Provision  for  redirecting  output  to  different  media. 

This  is  a  binary  metric  which  identifies  if  consideration  is  given 
to  the  caoability  to  redirect  output  to  different  media  dur'nq 
requirements  analysis,  design,  and  implementation. 


Software  System  Independence 

S3.1  Software  System  Independence  Measure  (design  and  implementation  onases 
at  system  level).  The  metric  is  the  sum  of  the  scores  of  the  followina  appi-:- 
able  elements  divioed  by  the  numper  of  applicable  elements. 


(1)  Dependence  on  Software  System  >.  tility  programs. 

The  more  utility  programs,  library  routines,  and  other  system 
facilities  that  are  used  within  a  system,  the  .more  dependent 
the  system  is  on  that  software  system  environment.  A  S0RT 
utility  in  one  operating  system  is  unlikely  to  be  exactly 
similar  to  a  SORT  utility  in  another.  This  measure  is  based 
on  the  number  of  references  to  system  facilities  in  a  module 
divided  by  the  total  number  of  lines  of  code  in  the  module. 

It  is  to  be  applied  during  design  and  implementation. 

(2)  Common,  standard  subset  of  language  used. 

The  use  of  nonstandard  constructs  of  a  language  that  may  be 
available  from  certain  compilers  cause  conversion  problems 
when  the  software  is  moved  to  a  new  software  system  environment. 
This  measure  represents  that  situation.  It  is  based  on  the 
number  of  modules  which  are  coded  in  a  non-standard  subset  of 
the  language.  The  standard  subset  of  the  language  is  to  be 
established  during  design  and  adhered  to  during  implementation. 


Machine  Independence 

MI.l  Machine  Independence  Measure  (design  and  implementation  at  system  levei) 
The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Programming  language  used  available  on  other  machines. 

This  is  a  binary  measure  identifying  if  the  programming  language 
used  is  available  (1)  on  other  machines  or  not  (0).  This  means 
the  same  version  and  dialect  of  the  language. 

(2)  Free  from  input/ output  references. 

Input  and  output  references  bind  a  module  to  the  current  macnine  con 
figuration.  Thus  the  fewer  modules  within  a  system  that  contain 
input  and  output  references,  the  more  localized  the  problem  Becomes 
when  conversion  is  considered.  This  measure  represents  that  fact 
and  is  based  on  the  number  of  I/O  references  within  a  module. 

It  is  to  be  applied  during  design  and  implementation. 

(3)  Code  is  independent  of  word  and  character  size  (implementation  only) 
Instructions  or  operations  which  are  dependent  on  the  word  or 
character  size  of  the  machine  are  to  be  either  avoided  or  param¬ 
etric  to  facilitate  use  on  another  machine.  This  measure  applied 

to  the  source  during  implementation  is  based  on  the  number  of 
modules  which  contain  violations  to  the  concept  of  independence  of 
word  and  character  size. 

' -1 )  Data  representation  machine  independent  (implementation  oni,  . 

The  naming  conventions  (length)  used  are  to  be  standard  or  com¬ 
patible  with  other  machines.  This  measure  is  based  on  ‘he  number 
of  modules  which  contain  variables  which  do  not  conform  to  standard 
data  representations . 


Communications  Commonai i tv 


CC.I  Communications  Coirroonality  Checklist  (ail  three  ohases  at  system 
level;.  The  metric  is  the  sum  or'  the  scores  of  the  following  applicable 
elements  divided  by  the  number  of  applicable  elements. 

(1)  Definitive  statement  of  requirements  for  corrmuncation  with  other 
systems  (requirements  only). 

During  the  requirement  phase,  the  communication  requirements 
with  other  systems  must  be  considered.  This  is  a  binary  measure  of 
the  existence  of  this  consideration. 

(2)  Protocol  standards  established  and  -oi lowed. 

The  cummui  cation  protocol  standards  for  conmuni cation  with  other 
systems  are  to  be  established  during  the  lesign  ohase  and  renewed 
durinq  implementation.  his  binary  measup  applied  at  each  of 
these  phases  indicates  whether  the  star cards  w er»  established  and 
followed. 

(3)  Single  module  inter-ao  j r  input  from  ir  •  . r. 

The  more  modules  which  nandle  inDut  :  jit  it  is  to 

interface  with  another  system  pc  jndard  orotocclj. 

This  measure  based  on  the  inverse  or  one  turps  of  modules  wnich 
handle  input  is  *o  oe  applied  to  the  iesion  specification  mo  sourc 
code. 

(<i)  Single  module  interface  *or  cutout  to  another  5/stem 

For  similar  reasons  as  ! 3)  above  this  measure  is  The  inverse  of 
the  number  of  output  modules. 

Data  Commonality 

DC.  1  Data  Commonality  Checklist  all  ‘nree  chases  at  system  level',  "e 
metr-c  is  the  sum  of  ‘he  scores  of  ‘.tie  ’Vlownc  aoDlicacle  elJmen",  5-  ,•  ten 
cy  tt  e  °umber  0  f  aooli  cable  elemen*-. . 


(1)  Definitive  statement  for  standard  data  representation  for  cofrcnun ; ca¬ 
tions  with  other  systems  ( requ i rements  only). 

This  is  a  binary  measure  of  tne  existence  of  consideration  for 
standard  data  representation  between  systems  which  are  to  be  interfaced. 
This  must  be  addressed  and  measured  in  the  requirements  phase. 

(2)  Translation  standards  among  representations  established  and  followed 
(design  and  implementation). 

More  than  one  translation  from  the  standard  data  representations  used 
for  interfacing  with  other  systems  may  exist  within  a  system.  Standards 
for  these  translations  are  to  be  established  and  followed.  This  binary 
measure  identifies  if  the  standards  are  established  during  design  and 
followed  during  implementation. 

(3)  Single  module  to  perform  each  translation  (design  and  implementation). 

For  similar  reasons  as  CC.l  (3)  and  (4)  above,  this  measure  is  the 
inverse  of  the  maximum  number  of  modules  which  perform  a  translation. 

Conciseness 

C0.1  Halstead's  Measure  (implementation  phase  at  module  level  first  then  system 
level).  The  metric  Is  based  on  Halstead's  concept  of  lenath  ( [HALSM77]) . 

The  observed  length  of  a  module  is 
N0  *  Nj  +  N2  where: 

Ni  *  total  usage  of  all  operators  in  a  module 
N2  *  total  usage  of  all  operators  in  a  module 

The  calculated  length  of  a  module  is 

Mg  *  njlog2nj  +  n2log2n?  where: 

*  number  of  unique  ODerators  in  a  module 
n2  *  number  of  unique  ooerato>-s  in  a  module 

The  metric  is  normalized  as  follows: 
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At  a  system  level  the  metric  is  the  averaoeo  value  of  il1  the  ■nodule  met 
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APPENDIX  D 

METRIC  ALGORITHMS 


(Refs  17,  23) 


METRIC  ALGORITHMS 


The  metric  scores  are  automatically  calculated  by  the  AMT  after 
tha  data  la  input  from  the  metric  worksheets.  Thie  appendix  lists 

i 

the  algorithms  uaed  to  oompute  the  metric  scores.  They  are  listed 
aocording  to  SDLC  phases  and  according  to  system  level  or  module  level 
application. 

RJMPIBBMEHTS  -  SYSTEM  LEVEL 

CP.l  -  ( WS 1.1.2  ♦  ((MSI. 1.3  -  WS 1.1.4) / WS 1 . 1.3)  + 

((MSI.  1.1  -  MS 1.1.5) /WS 1.1.1)  +  ( (MSI .1 . 1  -  MS1.I.6)/WS1.I.1)  ♦ 
MSI. 1.9  ♦  f MSI .1.1 1/MSI . 1.10) )/6 

AY.1  -  (MSI. II. 1  ♦  MSI .1 1 .2)/2 

ET.2  -  (MSI. II. 3) 

ET.2  -  (MSI. II. 4) 

ET.4  •  (MSI. II. 5) 

ET.5  «  (MSI. II. 6) 

AC . 1  »  (MSI. III. I  +  MSI. III. 2  +  MS  1 . 1 1 1 . 3)/3 
AA.l  -  (WS  1 .1 1 1 .4  «•  MSI  .II I  - 5 ) / 2 
0P.1  *  (WS1.IV.I  ♦  WS1.IV.2  *  WS  1 . 1 V .  3 )  /  3 
CM. 1  *  (WS1.IV.4) 

CM. 2  *  (WS1.IV.6  ♦  WS1.1V.5)/? 

CC.l  *  (WS1.IV.1) 


DC .1  -  (MSI. VI.?) 
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DESIGN  -  SYSTEM  LEVEL 


TR.1  -  WS2a. I . 1 

CP.l  -  ( ( (WS2a. I .2  -  WS2a. I .4)/WS2a. 1.2)  ♦  ( (WS2a. I .2  -  WS2a.I .3)/WS2a.I .2)  ♦ 

(WS2a.I .2  -  WS2a.I.5)/WS2a.I .2)  ♦  ((WS2a.I.6  -  WS2a.I .7)/WS2a.I .6) )/4 

AY . 1  *  WS2a. 1 1 . 1 

ET.l  -  (WS2a.II  .2  ♦  ( (WS2a. 1 1 .4  «■  WS2a  1 1 .5)/WS2a.  1 1 .3)  )/2 
ET.4  -  WS2a.1 1 .6 
ET.5  «  HS2a.II .7 

SI . 1  «  (WS2a.III. 1  +  { l/WS2a.IX. 1)  +  (WS2a.IX.3/WS2a.IX.l))/3 
If  HS2a.IX.1  -  0  Set  SI.1  to  Value  of  WS2a.III.l 

MO. 2  -  (1  -  (WS2a. Ill .4/US2a. 1 1 1 . 2) ) 

GE.l  -  (WS2a. 1 1 1 .4/WS2  a. 1 1 1 . 2 ) 

IN. 1  -  ((WS2a.VIII.4/WS2a.VIII.3)  +  (HS2a. VI 1 1 .2/HS2a. VI II . 1) )/2 

IN. 2  »  ((WS2a.VIII.6/W$2a.VIII.5)  ♦  (WS2a.VIII.8/WS2a.VIII.7))/2 

IN. 3  *  ( (WS2a. VIII. 10/WS2a.VI II .9)  +  WS2a. VI 1 1 . 1 1  )/2 

EE. 2  »  ( ( l/WS2a.IV.8)  ♦  HS2a.VI.7  +  WS2a.IV.4  + 

( (WS2a. IV.9  -  WS2a.IV.10)/WS2a.IV.9))/4 

j 

t 

, 


OESIGN  -  SYSTEM  LEVEL 


EE. 3  »  WS2a.IV.6 

SE.l  =  (WS2a. I V. 1  ♦  WS2a. IV. 2  ♦  WS2a.IV.3)/3 
AC.l  -  (WS2a.V . 1  ♦  WS2a.V .2  +  WS2a.V.3))/3 
AA.l  »  WS2a.V.4 

OP.1  •  (WS2a.VII.1  +  WS2a.VI I .9  ♦  WS2a.VII.10  ♦ 

WS2a.VII.6  ♦  (1  -  (WS2a.V!I.8/WS2a.VlI.7)))/5 

TN.l  -  WS2a.VI I . 13 

CM. 1  «  ( (WS2a. VI 1 . 16/WS2a. VI 1.15)  ♦  ( 1/HS2a.VII . 14)  + 

(1  -  (WS2a.VII.17/WS2a.VII.15))  *  WSa.VII.18  + 

WS2a.VI 1 . 19  ♦  WS2a. VI 1 ,20)/6 

CM. 2  -  (WS2a.VI I .21  +  WS2a.VII.22  ♦  WS2a.VII.23  +  ( l/WS2a.VI I .24)  ♦ 
WS2a.VII.25  +  WS2a.VII.26  +  WS2a. VI 1 ,27)/7 

CC.l  *  (WS2a.VI .2  +  WS2a.VI .3  +  ( l/WS2a.VI .4) )/3 

OC.l  «  (WS2a.VI .5  +  WS2a. VI .6  +  l/WS2a.VI .7)/3 


Til 


IMPLANTATION  -  SYSTEM  LEVEL 

CP.l  *  ((1  -  (WS2a.I.4/WS2a.I.2))  +  (1  -  (WS2a. I . 3/WS2a. 1.2))  ♦ 

( 1/WS2a. I .5) )/3 

ET.l  *  (WS2a. 1 1 .2  ♦  ( (WS2a. 1 1 .4  +  WS2a. 1 1 .5)/WS2a. 1 1 .3) )/2 
ET.4  «  WS2a.I I .6 
ET.5  *  WS2a.II.7 

SI . 1  «  ( (WS2a.IX.2/WS2a.IX. 1)  +  ( 1/WS2a. IX. 1 ) )/2 
MO. 2  -  (1  -  WS2a.1 1 1 .4/wS2a.1 1 1.2) 

6E.1  «  (WS’a.III  .4/VIS2a.III  .2) 

IN. I  -  ((WS2a. VIII. 2/WS2a. VIII. 1)  +  ( WS2a. VI 1 1 .4/WS2a. VI 1 1 .3) )/2 
IN.:  =  ((WS2a.VIII.6/WS2a.Vi:i.5)  +  (WS2a.VI 1 1 .8/WS2a.VI 1 1 .7) )/2 
IN. 3  *  ( (WS2a.VI II . 10/WS2a.VI 1 1 .9)  +  WS2a.VIII.ll)/2 
EE. 2  «  (( l/WS2a.IV.8)  ♦  ((WS2a.IV.10  -  WS2a.IV.  1 1  )/WS2a.  IV.9) )/?. 

EE. 3  *  WS2 a. IV .6 

SE.l  *  (WS2a. IV. 2  ♦  WS2a.IV.5  ♦  (1  -  (WS2a. I V . 10/WS2a. I V .9) )  ♦  HS2a.IV.3)/4 


AC.l  *  (WS2a. V . 1  ♦  WS2a.V.2  ♦  WS2a.V.3)/3 


IMPLEMENTATION  -  SYSTEM  LEVEL 


AA.l  »  WS2a.V.4 

OP.l  -  (WSZa.VII.l  +  WS2a.VI 1.9  +  WS2a.VII.10  ♦  (1  -  WS2a.VII .3/WS2a.VI 1.4)  ♦ 
WS2a. VI 1 .5  ♦  WS2a.VI 1 .6  ♦  (1  -  (WS2a.VI 1 .8/WS2a.VI 1 .7) ) )/7 

TN.l  -  (WS2a.VII.Il  +  WS2a.VI 1 . 12  ♦  WS2a. VI 1 . 1 3 ) / 3 

CM. 1  -  ( (WS2a.VI I . 16/WS2a.VI I . 15  ♦  ( 1/WS2a.VII . 14)  + 

(WS2a.VII.17/WS2a.VII. 15)  +  WS2a.VII.18  +  WS2a.VII.19  ♦  WS2a. VI 1 .20)/6 

CM. 2  ■  (WS2a.VII .21  ♦  WS2a.VII.22  +  WS2a.VII.23  ♦  ( l/WS2a.VI I .24)  ♦ 

WS2a.  VII. 25  +  WS2a.VII.26  ♦  WS2a.VI I . 27 )/7 

CC.l  •  ( ( l/WS2a.VI . 1)  ♦  WS2a.VI .2  ♦  WS2a.VI.3  ♦  (1/WS2a.VI.4))/4 

OC.1  •  (WS2a. VI .5  ♦  WS2a. VI .6  ♦  ( l/WS2a.VI .7) )/3 
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DESIGN  -  MODULE  LEVEL 


CP.l  -  (WS2b. 1 . 1  ♦  ( l/WS2b. 1 .2)  ♦  WS2b.I.3  +  (1  -  WS2b. 1 .6/WS2b. 1 .4)  )/4 
CS.l  -  (WS2b. VI 1 1 . 1  ♦  WS2b.VI 1 1 .2  +  WS2b.VIII.3  +  WS2b.VI 1 1 .4)/4 
CS.2  -  (WS2b.VI 1 1 .5  +  WS2b. VI II . 6 ) / 2 
AY . 1  ■  WS2b.I I .2 
ET.l  -  WS2b.II . 1 

ET.2  -  (WS2b.II .3  +  WS2b. 1 1 .4  ♦  WS2b.II.5  +  WS2b.II.6)/4 
ET.3  ■  (WS2b.I I .7  +  WS2b.II.8  +  WS2b.II.9)/3 
SI.l  «  (WS2b. 1 1 1 .5  +  WS2b. 1 1 1 .6)/2 
SI. 3  -  l/WS2b.1 1 1 . 1 

MO. 2  ■  ( (WS2b.IV.4/WS2b.IV.3)  +  WS2b.IV.5  ♦ 

WS2b.IV.6  ♦  WS2b.IV.7  ♦  WS2b.IV.8)/5 

GE.2  -  ((1  -  WS2b.IV.9)  +  ( l/WS2b. IV. 10)  + 

WS2b.IV.Il  ♦  WS2b.IV.12)/4 

EX.l  «  US2b.V.  1 

EX. 2  »  (WS2b.V .2  +  WS2b.V.3)/2 
EE . 1  *  WS2b.VI . 1 
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DESIGN  -  MODULE  LEVEL 

EE. 2  -  (WS2b.VI .3  +  WS2b.VI.4)/2 
EE. 3  -  WS2b.VI .5 

SS. 1  -  ( ( 1/WS2b.IV. 1)  +  WS2b.IV. 13)/2 
MI . 1  -  (WS2b.IV. 14  +  1/WS2b.IV.2)/2 

IMPLEMENTATION  -  MODULE  LEVEL 

TR.T  *  WS3.III.4 

CP.l  -  (WS2b.I . 1  ♦  l/WS2b. I .2  ♦  WS2b.I.3  ♦  (1  -  WS2b.I .6/WS2b.I .4) )/4 

CS.l  «  (WS2b.VI 1 1 .2  +  WS2b. VI 1 1 .3  ♦  HS2b.VI 1 1 .4)/3 

CS.2  -  ( WS2b . V 1 1 1 . 5  ♦  1/WS3.VI . 3 ) / 2 

AY . 1  »  (WS2b.1 1 .2  +  WS3 .X 1 . 1 )/2 

ET.l  -  WS3.VII .3 

ET.2  -  (MS3.IV.4  ♦  WS3.IV.5  ♦  WS3.IV.6)/3 
ET.3  «  ( ( 1/WS3.VI 1 . 1)  +  WS3.VII.2  ♦  WS3.VII.4)/3 


on;, 


IMPLEMENTATION  -  MODULE  LEVEL 


51.1  «  (WS2b.1 1 1 .5  ♦  WS2b.1 1 1 .6  ♦  WS2b.III.7  ♦ 

( l/(WS2b. 1 1 1 .3  ♦  WS2b. 1 1 1 .9) ) )4 

51. 3  *  1/US3.I . 10 

51. 4  «  (WS3.I.20  +  (1  -  (WS3.I .18/WS3.I .2  -  WS3.I.4)))  + 

(1  -  (WS3.1 . 15/WS.1 . 14) )  +  (1  -  (WS3.I . I6/WS3.1 . 14) )  ♦ 

(1  -  (WS3.I . 17/WS3.I .2) )  +  (1  -  (WS3.I .6/(WS3.I .2  -  WS3.I.4)))  + 
1/WS3.1 .9  ♦  (1  -  ( (WS3.1 . 10  -  WS3.1 . 1 1)/(WS3. 1 .2  -  MS3.I.4)))  ♦ 

(1  -  ( (WS3.I . 12  -  WS3.I . 13)/(WS3.I .2  -  WS3.I.4)))  ♦ 

(WS3.VI . 1 / (WS3 .VI . 1  +  WS3.VI .2) )  ♦ 

(1  -  ( (WS3.VI . 1  ♦  US3.VI .2)/(WS3.1 .2  -  W$3. 1 .4) ) ) )/ 1 1 

MO. 2  -  (WS3.I.1  +  WS3.V.5/WS3.V.3  ♦  1/WS3.V.4  ♦ 

1/WS3.V.6  +  WS3.V.7  +  WS3.V.8  ♦  WS3.V.9  ♦  WS2b.IV.8)/8 
For  Imo3 .1.1  _  100  then  0 
For  WS3.I.1  100  then  1 

6E.2  -  (WS2b.IV.9  +  1/WS3.IX.2  ♦  WS3.X.2  +  WS3.X3)/4 
If  WS3.IX.2  ■  0  Set  2nd  term  to  1 

EX.l  *  (WS2b.V. 1  ♦  (WS3 .X 1 .7/(WS3 .X 1 .4  +  WS3.XI.5  +  WS3.XI.6  ♦  WS3. .XI . 7) ) )/2 
EX. 2  «  (WS3.X.4  ♦  WS3.X.1  +  (WS3.XI.5  -  WS3.XI .4)/WS3.XI .5) )/3 

50.1  *  US3.II I .2/WS2.I .2 

If  SO.l  1  Set  to  1 
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IMPLEMENTATION  -  MODULE  LEVEL 


SD.2  «  (WS3.III.3  +  WS3.III.il  ♦  1/WS3.III.5  ♦ 

WS3.II1.6  +  WS3.III.7  ♦  1/WS3 .111.8  +  WS3. 1 1 1 . 10)/7 

SD.3  «  ((1  -  WS3.I .3/US3.I . 1)  ♦  WS3.III.9  +  WS3.III.il  ♦ 

(1  -  WS3 .III. I2/WS3 .1.1) )/4 

EE. 2  »  1  -  (WS3.VI 1 1 .3/WS3 .1.14) 

EE. 3  *  ( (WS3.VI II .2/(WS3.VI . 1  ♦  WS3.VI.2))  + 

(1  -  WS3.VIII .  1/ ( WS3 . 1 .2  -  WS3.I.4))  ♦ 

1/WS3.VI.4  +  WS2b. VI .5  +  {1  -  (WS3.XI .9/(WS2a. IX . 1 ) ) )/5 

SS.l  -  ( (WS3.V.2/(WS3.I .2  -  WS3.I.4))  ♦  WS2b.IV . 13)/2 

MI . 1  *  (WS2b. IV . 14  +  (1  -  ( (WS3.IV.  1  ♦  WS3.IV.2)/(WS3.1 .2  -  WS3.I.4)))  + 
WS3.IX.1  + 

(1  -  (WS3.IX.2/(WS3.I.2  -  WS3.I.4)))  +  WS3.IX.3)/5 


DEFINITION  OF  QUALITY  ATTRIBUTES 


The  software  quality  attributes  (factors  or  criteria) 
contained  in  this  appendix  are  found  in  the  literature. 
The  authors  of  the  quoted  or  paraphrased  definitions  are 
indicated  in  parentheses. 


ACCEPTABILITY 

how  closely  the  ADP  system  meets  true  user  needs  (Rosy) 

measure  of  how  closely  the  computer  program  meets  the 
true  needs  of  the  user  (SAMSO) 

relates  to  degree  to  which  software  meets  the  user's 
needs  including  the  clarity  and  unambiguity  of  the 
specifications  and  the  effectiveness  of  the  man-machine 
interface  (USA  ISRAD) 

-  does  the  software  meet  the  need  of  the  user  (Light) 


ACCESSIBILITY 

extent  that  software  facilitates  the  selective  use  of 
its  components  (Boehm) 

those  attributes  of  software  that  provide  for  an  audit 
and  control  of  the  access  of  software  and  data  (McCall) 


ACCOUNTABILITY 

extent  that  code  usage  can  be  measured  (Boehm) 


ACCURACY 

the  degree  of  exactness  of  the  data  contained  in  a 
product  unit  (Cho) 

where  mathematically  possible  a  routine  shou Id  give  an 
approximation  that  is  as  close  as  practicable  to  the 
full  machine  precision  for  whatever  machine  it  is 
running  on  ( Schon fe lder  ) 
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extent  that  its  outputs  are  sufficiently  precise  to 
satisfy  its  intended  use  (Boehm) 


the  mathematical  calculations  are  correctly  per¬ 
formed  (Rubey) 

measure  of  the  quality  of  freedom  from  error,  degree  of 
exactness  possessed  by  an  approximation  or  measure¬ 
ment  (Gilb) 


ADAPTABILITY 

how  much  time  and  effort  are  required  to  modify  a  soft¬ 
ware  system  (Kosy) 

-  measure  of  the  ease  with  which  a  program  can  be  altered 
to  fit  differing  user  images  and  system  con¬ 
straints  (Poole) 

measure  of  the  ease  of  extending  the  product,  such  as 
adding  new  user  functions  to  the  product  (Myers) 

a  measure  of  the  effort  required  to  modify  a  computer 
program  to  add,  change  or  remove  a  function  or  use  the 
computer  program  in  a  different  environment  (includes 
concepts  of  flexibility  and  portability)  (SAMSO) 

relates  to  ability  of  software  to  operate  inspite  of 
unexpected  input  or  external  conditions  (USA  iSRAD) 


AVAILABILITY 

fraction  of  total  time  during  which  the  system  can 
support  critical  functions  (SADPR-85) 

-  error  recover  and  protection  (Liskov) 

-  probability  that  a  system  is  operating  satisfactorily 
at  any  point  in  time,  when  used  under  stated  condi¬ 
tions  (Gilb) 


COMMUNICATIVENESS 

extent  that  software  facilitates  the  specifications  of 
inputs  and  provide  outputs  whose  form  and  content  are 
easy  to  assimilate  and  useful  (Boehm) 


COMPATABILITY 

measure  of  portability  that  can  be  expected  of  systems 
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when  they  are  moved  from  one  given  environment  to 
another  (Gilb) 


COMPLETENESS 

-  extent  to  which  software  fulfills  overall  mission 
satisfaction  (McCall) 

extent  that  all  of  its  parts  are  present  and  each  of  its 
parts  are  fully  developed  (Boehm) 


COMPLEXITY 

relates  to  data  set  relationships,  data  structures, 
central  flow,  and  the  algorithm  being  imple¬ 
mented  (Richards) 

measure  of  the  degree  of  decision-making  logic  within 
a  system  (Gilb) 


CONCISENESS 

the  ability  to  satisfy  functional  requirements  with  a 
minimum  amount  of  software  (McCall) 

extent  that  no  excessive  information  is  present  (Boehm) 


CONSISTENCY 

degree  to  which  software  satisfies  specifica¬ 
tions  (McCall) 

-  extent  that  it  contains  uniform  notation,  terminology, 
and  symbology  within  itself  and  the  extent  that  the 
content  is  traceable  to  the  requirements  (Boehm) 


CONVERTIBILITY 

degree  of  success  antici;  ated  in  readying  people, 
machines,  and  procedures  to  support  the  system 
.  (SADPR-85) 


CORRECTNESS 

correctness  of  its  description  with  respect  to  the 
objective  of  the  softwars  as  specified  by  the  semantic 
description  of  the  linguistic  level  it  defines  (Pernio ) 


the  coding  of  a  computer  program  module  ^hirh  properly 


and  completely  implements  selected  overall  system 
requirements  (NSSC  PATHWAY) 

relates  to  degree  to  which  software  is  free  from  design 
and  program  defects  (USA  ISRAD) 

the  program  is  logically  correct  (Rubey) 


COST 

includes  not  only  development  cost,  but  also  the  costs 
of  maintenance,  training,  documentation,  etc.,  on  the 
entire  life  cycle  of  the  program  (Wulf) 

there  are  three  major  categories  of  cost: 

Economy  of  operation  -  relates  to  cost  or  operating 
system 

Economy  of  modification  -  relates  to  cost  of  making 
changes  to  software  to  meet  new  requirements  or  correct 
defects  resulting  in  errors  in  requirements,  design, 
and  programming 

Economy  of  development  -  relates  to  cost  of  entire 
development  cycle  from  identification  of  requirement  to 
initial  operation 

-  development  and  maintenance  costs  (Myers) 

implementation  cost  and  operational  cost  (Gilb) 


DOCUMENTATION 

quality  and  quantity  of  user  publications  which  provide 
ease  of  understanding  or  use  (Myers) 


EFFICIENCY 

capability  to  accomplish  functions  with  minimum 
resources  (Cho) 

measure  of  the  execution  behavior  of  a  program  (execu¬ 
tion  speed,  storage  speed)  (Myers) 

execution  time,  storage  space,  #  instructions,  pro¬ 
cessing  time  (Rosy) 

extent  that  software  fulfills  its  purpose  without  waste 
of  resources  (Boehm) 

the  ratio  of  useful  work  performed  to  the  total  energy 
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expended  (Gilb) 

the  amount  of  computing  resources  and  code  required  by  a 
program  to  perform  a  function  (McCall) 

reduction  of  overall  cost  -  run  time,  operation,  mainte¬ 
nance  (Kernighan) 

extremely  fast  run  time  and  efficient  overlay  capa¬ 
bilities  (Richards) 

computation  time  and  memory  usage  are  optimized  (Rubey) 


EXPANDABILITY /FLEXI BILITY/AUGMENTABILITY 

attributes  of  software  that  provide  for  expansion  of 
data  storage  requirements  or  computational 
functions  (McCall) 

extent  to  which  system  can  absorb  workload  increases 
and  decreases  which  require  modification  (SADPR-85) 

the  ability  of  a  system  to  immediately  handle  different 
logical  situations  (Gilb) 

how  easily  the  software  modules  comprising  the  system 
or  subsystem  can  be  rearranged  to  change  the  system's 
functions  without  modifying  any  of  the  modules  (Rosy) 

ease  of  changing,  expanding,  or  upgrading  a 
program  (Yourdon) 

the  software  modules  must  be  usable  in  a  variety  of 
contexts  (Culpepper) 

includes  changeability?  e.g.,  the  ease  of  correction 
bugs,  maintenance  because  of  changing  specifications, 
and  portability  to  move  to  another  system  (Ledgard) 

extent  that  software  easily  accommodates  expansions  in 
data  storage  requirements  or  component  eomputat ional 
functions  (Boehm) 

-  attributes  of  software  which  allow  quick  response  to 
changes  in  algorithms  (Richards) 

ability  to  reuse  the  software  and  transfer  it  to  another 
processor  (includes  reuse,  adaptability,  transferability 
and  versatility  of  software)  (Light) 


EXPRESSION 

how  a  program  is  expressed  determines  in  large  measure 


the  intelligibility  of  the  whole  (Kernighan) 


EXTENSIBILITY 

extent  to  which  system  can  support  extensions  of  criti¬ 
cal  functions  (SADPR-85) 


GENERALITY /REUSABILITY 

those  attributes  of  software  that  provide  breadth  to  the 
functions  performed  (McCall) 

measure  of  the  scope  of  the  functions  that  a  program 
performs  (Myers) 

building  programs  from  reusable  software  modules  can 
significantly  reduce  production  costs  (Goodenough) 

-  how  broad  a  class  of  similar  functions  the  system  can 
perform  (Rosy) 

standardized  modules  can  be  lifted  from  one  program  and 
used  in  another  without  extensive  recoding  or  retest¬ 
ings  (Whipple) 

-  degree  to  which  a  system  is  applicable  in  different 
environments  (Gilb) 


HUMAN  FACTORS 

system  should  be  easy  to  use  and  difficult  to 
misuse  (Cho 

every  program  presents  an  interface  to  its  human  users./ 
operators,  by  human  factors  we  refer  collectively  to  all 
the  attributes  that  make  this  interface  more  palatable: 
ease  of  use,  error  protectedness,  quality  of  documenta¬ 
tion,  uniform  syntax,  etc.  (Wulf) 

extent  that  software  fulfills  its  purpose  without 
wasting  user's  time  and  energy  or  degrading  their 
morale  (Boehm) 

measure  of  the  product's  ease  of  understanding,  ease  of 
use,  difficulty  of  misuse,  and  frequency  of  user's 
errors  (Myers) 


INTEGRITY 

extent  to  which  access  to  software  or  data  by  unathor- 
ized  persons  can  be  controlled  (McCall ) 
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how  much  the  operation  of  one  software  subsystem  can 
protect  the  operation  of  another  (Rosy) 

a  measure  of  the  degree  of  protection  the  computer 
program  offers  against  unauthorized  access  and  loss  due 
to  controllable  events  (includes  the  concepts  of 
privacy  and  security)  (SAMSO) 

relates  to  ability  of  software  to  prevent  purposeful 
or  accidental  damage  to  the  data  or  software  (USA  ISRAD) 

-  ability  to  resist  faults  from  personnel,  the  security 
problem,  or  from  the  environment,  a  fault  tolerance 
issue  (Light) 

probablity  of  system  survival  when  subjected  to  attack 
during  a  time  interval  (Gilb) 


INTEROPERABILITY 

effort  required  to  couple  one  system  with 
another  (McCall) 

how  quickly  and  easily  one  software  system  can  be 
coupled  to  another  (Rosy) 

relates  to  how  quickly  and  easily  one  software  system 
can  be  coupled  to  another  (USA  ISRAD) 


LEGIBILITY 

-  extent  that  its  functions  and  those  of  its  components 
statements  are  easily  discerned  by  reading  the 
code  (Boehm) 


MAINTAINABILITY 

ability  to  keep  up  with  its  intended  use  (Cho) 

measure  of  the  effort  and  time  required  to  fix  bugs  in 
the  program  (Myers) 

how  easy  it  is  to  locate  and  correct  errors  found  in 
operational  use  (Rosy) 

extent  that  the  software  facilitates  updating  to  satisfy 
new  requirements  (Boehm) 

maintenance  involves 

(1)  correction  to  heretofore  latent  bug 
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(Lieblein ) 


(2)  enhancements 

(3)  expansion 

(4)  major  redesign 

ease  with  which  a  change  can  be  made  due  to 

(1)  bug  during  operation 

(2)  non-satisfaction  of  users  requirements 

(3)  changing  requirements 

(4)  obsolescence/upgrade  of  system  (McCall) 

probability  that  a  failed  system  will  be  restored  to 
operable  condition  within  a  specified  time  (Gilb) 


MANAGEABILITY 

-  degree  to  which  system  lends  itself  to  efficient 
administration  of  its  components  (SADPR-85) 


MODIFIABILITY 

change  and  enhancement  of  the  system  should  be  easily 
implemented  (Cho) 

measure  of  the  cost  of  changing  or  extending  the 
program  (Myers) 

operational  experience  usually  shows  the  need  for 
incremental  software  improvements  (Goodenough) 

-  extent  that  it  facilitates  the  incorporation  of 

changes,  once  the  nature  of  the  desired  change  has  been 
determined  (Boehm) 

quality  of  a  program  that  reduces  the  effort  required 
to  alter  it  in  order  to  conform  to  a  modification  of  it 
specification  (Wulf ) 

internal  (detailed  design)  characteristics  of  a  program 
module  are  arranged  so  as  to  permit  easy  change 
( NSSC  PATHWAY) 

use  of  HOL  reduces  programmer's  task  and  human  errors 
and  allows  smaller  units  to  be  tested  permitting  easier- 
debugging  (Whipple) 

the  program  is  easy  to  modify  (Rubey) 


MODULARITY /STRUCTUREDNESS 

extent  to  which  a  system  uses  program  -onstructs  for 
better  readability  (Cho) 
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ability  to  combine  arbitrary  program  modules  into  larger 
modules  without  knowledge  of  the  construction  of  the 
modules  (Goos) 

the  software  must  consist  of  modules  with  well  defined 
interfaces.  Interactions  between  modules  must  occur 
only  at  those  interfaces  (Culpepper) 

extent  that  it  possesses  a  definite  pattern  of  organiza¬ 
tion  of  its  independent  parts  (Boehm) 

how  well  a  program  is  organized  around  its  data  repre¬ 
sentation  and  flow  of  control  (Kernighan) 

-  there  is  no  interference  between  program  entities 
(Rubey ) 

formal  way  of  dividing  a  program  into  a  number  of  sub 
units  each  having  a  well  defined  function  and  relation¬ 
ship  to  the  rest  of  the  program  (Mealy) 


PERFORMANCE 

the  effectiveness  with  which  resources  of  the  host 
system  are  utilized  toward  meeting  the  objective  of  the 
software  system  (Dennis) 

refers  to  such  attributes  as  size,  speed,  precision; 
e.g.,  the  rate  at  which  the  program  consumes  accountable 
resources  (Wulf) 


PORTABILITY /TRANSFERABILITY 

programs  must  be  readily  transferable  among  different 
equipment  configurations  (Goodenough/Culpepper ) 

degree  of  transportability  is  determined  by  number, 
extent,  and  complexity  of  changes,  and  hence  the  diffi¬ 
culty  in  implementing  a  software  processor  which  can 
mechanically  move  a  program,  between  a  specified  set  of 
machines  (Hague) 

machine  -  independence  (Marshall) 

measure  of  the  ease  of  moving  a  computer  program  from 
one  computing  environment  to  another  (Meahy,  Poole) 

how  quickly  and  cheaply  the  software  system  can  be  con¬ 
verted  to  perform  the  same  functions  using  different 
equipment  (Kosy) 

an  appropriate  environment  can  be  provided  on  most 
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computers  (Goos) 


extent  that  it  can  be  operated  easily  and  well  on 
computer  configurations  other  than  its  current  one 
(Boehm) 

moving  software  from  one  computer  (environnent )  to 
another  (Lieblein) 

-  measure  of  the  effort  required  to  install  a  program  on 
another  machine,  another  operating  system,  or  a 
different  configuration  of  the  same  machine  (Wulf) 

external  (black  box)  form,  fit,  and  function  character¬ 
istics  of  a  program  module  which  permit  its  use  as  a 
building  block  in  several  computer  programs 
(NSSC  PATHWAY) 

relates  to  how  quickly  and  easily  a  software  system  can 
be  transferred  from  one  hardware  system  to  another 
(USA  ISRAD) 

ease  of  conversion  of  a  system  from  one  environment  to 
another;  the  relative  conversion  cost  for  a  given  con¬ 
version  method  (Gilb) 

effort  required  to  transfer  a  program  from  one  hardware 
configuration  and/or  software  system  environment  to 
another  (McCall) 


PRECISION 

the  degree  to  which  calculated  results  reflect 
theoretical  values  (Gilb) 


PRIVACY 

the  extent  to  which  access  to  sensitive  data  by  unautho¬ 
rized  persons  can  be  controlled  and  the  extent  to  which 
the  use  of  the  data  once  accessed  can  be  con¬ 
trolled  (McCall) 

relates  to  the  protection  level  for  data  in  the  system 
and  the  individual's  right  to  review  and  control 
dissemination  of  data  (USA  rSRAD) 


RELIABILITY 

software's  ability  to  perform  what  it  is  supposed  to  do 
under  defined  conditions  (Cho) 

includes  correctness,  testing  for  errors,  and  error 
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tolerance  ( Ledgard ) 


-  the  probability  that  the  software  will  satisfy  the 
stated  operational  requirements  for  a  specified  time 
interval  or  a  unit  application  in  the  operational  en¬ 
vironment  ( JLC  SRWG ) 

the  probability  that  a  software  system  will  operate 
without  failure  for  at  least  a  given  period  of  time  when 
used  under  stated  conditions  (Kosy) 

extent  to  which  a  program  can  be  expected  to  perform  its 
intended  functions  satisfactorily  (Thayer) 

ability  of  the  software  to  perform  its  functions 
correctly  in  spite  of  failures  of  computer  system 
components  (Dennis) 

-  probability  that  a  software  fault  does  not  occur  during 
a  specified  time  interval  (or  specified  number  of  soft¬ 
ware  operational  cycles)  which  causes  deviation  from 
required  output  by  more  than  specified  tolerances,  in  a 
specific  environment  (Thayer) 

measure  of  the  number  of  errors  encountered  in  a 
program  (Myers) 

extent  to  which  a  program  can  be  expected  to  perform  its 
intended  function  with  required  precision  (McCall) 


REPAIRABILITY 

-  probability  that  a  failed  system  will  be  restored  to 
operable  condition  within  a  specified  active  repair 
time  when  maintenance  is  done  under  specified  condi¬ 
tions  (Gilb) 


ROBUSTNESS 

ability  to  continue  execution  under  certain  imperfect 
conditions  (Cho) 

routines  should  be  coded  so  that  when  it  is  not  possible 
to  return  a  result  with  any  reasonable  accuracy  or  there 
is  a  danger  of  causing  some  form  of  machine  failure  they 
should  detect  this  and  take  appropriate  actions 
( Schonf elder ) 

quality  of  a  program  that  determines  its  ability  to 
continue  to  perform  despite  some  violation  of  the 
assunptions  in  its  specifications  (Wulf) 

program  should  test  input  for  plausability  and 


validity  (Kernighan) 


SECURITY 

the  ability  to  prevent  unauthorized  access  to  programs 
or  data  (Rosy) 

-  extent  to  which  access  to  software,  data,  and  facilities 
can  be  controlled  (SADPR-85) 

-  measure  of  the  probability  that  one  system  user  can 
accidentally  or  intentionally  reference  or  destroy  data 
that  is  the  property  of  another  user  or  interface  with 
the  operation  of  the  system  (Myers) 

-  relates  to  the  ability  of  the  software  to  prevent 
unauthorized  access  to  the  system  or  system  elements 
(USA  ISRAD) 


SELF-CONTAINEDNESS 

-  extent  to  which  a  program  performs  all  its  explicit  and 
implicit  functions  within  itself  (Boehm) 


SELF-DESCRIPTIVE NESS/CLARITY 

-  those  attributes  of  software  that  provide  explanation 
of  the  implementation  of  a  function  (McCall) 

-  measure  of  how  clear  a  program  is,  i.e.,  how  easy  it  is 
to  read,  understand,  and  use  (Ledgard) 

refers  to  the  ease  with  which  the  program  (and  its 
documentation)  can  be  understood  by  humans  (Wulf) 

extent  that  it  contains  enough  information  for  a  reader 
to  determine  its  objectives,  assunqptions,  constraints, 
inputs,  outputs,  components,  and  status  (Boehm) 


SERVICEABILITY 

-  degree  of  ease  or  difficulty  with  which  a  system  can  be 
repaired  (Gilb) 


STABILITY 

-  the  "ripple  effect"  or  how  many  modules  have  to  be 
changed  when  you  make  a  change  (Myers) 

-  measure  of  the  lack  of  perceivable  change  in  a  system 
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in  spite  of  the  occurrence  in  the  environment  which 
would  normally  be  expected  to  cause  a  change  (Gilb) 


TESTABILITY 

the  ease  of  checking  the  quality  of  system  output 
through  use  of  messages  (Cho) 

instrumentation  and  debugging  aids  (Liskov) 

minimize  testing  costs  (Yourdon) 

provision  of  facilities  in  the  design  of  programs  which 
are  essential  to  testing  complex  structures  (Edwards) 

extent  that  software  facilitates  the  establishment  of 
acceptance  criteria  and  supports  evaluation  of  its 
performance  (Boehm) 

a  measure  of  the  effort  required  to  exercise  the  com¬ 
puter  program  to  see  how  well  it  performs  in  a  given 
environment  and  if  it  actively  solves  the  problem  it 
was  supposed  to  solve  ( SAMSO  ) 

-  effort  required  to  test  a  program  to  insure  it  performs 
its  intended  function  (McCall) 

-  measure  of  our  ability  to  test  software  (Light) 


TIME 

two  major  categories  of  time: 

Modification  Time  -  relates  to  total  elapse  time  from 
point  when  new  requirement  or  modification  is  identi¬ 
fied  the  change  is  implemented  and  validated 

Development  Time  -  relates  to  total  elapsed  time  of 
development  (USA  ISRAD) 

development  time  (Myers) 

what  is  the  expected  life  span  of  system  (Gilb) 


TOLERANCE 

measure  of  the  systems  ability  to  accept  different  forms 
of  the  same  information  as  valid  or  withstand  a  degree 
of  variation  in  input  without  malfunction  or  rejec¬ 
tion  (Gilb) 


UNDERSTANDABILITY 


meaningful  variable  names,  brief  comments,  traceable 
module  interface  and  data  flow  (Cho) 

-  ease  with  which  the  implementation  can  be  under¬ 
stood  (Richards) 

reduced  complexity,  reduced  redundancy,  clear 
documentation/notation  (Goodenough ) 

extent  that  the  purpose  of  the  product  is  clear  to  the 
evaluator  (Boehm) 

documentation  remains  current  (Whipple) 
the  program  is  intelligible  (Rubey) 

UNIFORMITY 

module  should  be  usable  uniformly  (Goodenough) 


USABILITY /OPERABILITY 

-  effort  required  to  learn,  operate,  prepare,  input,  and 
interpret  output  of  a  program  (McCall) 

measure  of  the  human  interface  of  the  program  (Myers) 

ease  of  operation  from  human  viewpoint,  covering  both 
human  engineering  and  ease  of  transition  from  current 
operation  (SADPR-85) 

how  suitable  is  it  to  the  use  (Rosy) 

-  software  must  be  adequately  documented  so  that  it  can 
be  easily  used  and  maintained  (Culpepper) 

extent  that  it  is  convenient  and  practicable  to  use 
(Boehm) 

the  program  is  easy  to  learn  and  use  (Rnbey) 


VALIDITY 

relates  to  degree  to  which  software  implements  the 
user's  specifications  (USA  1SRAD) 


APPENDIX  P 

SOFTWARE  SYSTEM  DEVELOPMENT  LIFE  CYCLE  FOR  AFLC/LM 


(Ref  89) 


Figure  16 
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Reviews 


Purpose  of  Reviews.  The  purpose  of  a  AFR  300-15  review  is  to  give  a  project  manager 
(t>M)  and  deputy  project  manager  (DPM)  a  means^of  assessing  and  verifying  the  degree  ol 
completion  of  tasks  related  to  milestones  and  a,  means  of  controlling  resources.  When  a 
review  has  been  completed  the  PM /DPM  can  give  management  the  information  on  which 
to  base  a  decision  to  continue,  redirect,  or  end  a  project.  While  all  reviews  have  the 
common  purpose  of  "assessing  and  verifying,"  they  also  have  more  specific  umt^ie 
purposes.  An  explanation  of  the  unique  purpose  of  each  review,  as  derived  from  AF 
regulations  and  experience  follows: 

a.  System  Requirements  Review  (SRR). 

(1)  To  make  sure  the  requirements  documented  in  the  Functional  Description 
(FD)  by  the  functional  OPR  (proponent),  which  are  to  be  satisfied  by  a  data  system(s), 
are  clear,  complete,  correct  and  consistent  with  those  in  the  validated  and  approved 
Data  Automation  Requirement  (DAR). 

(2)  To  make  sure  the  functional  OPR  and  data  automation  OPR  have  a  mutual 
understanding  of  the  requirements  in  the  FD  which  are  to  be  satisfied  by  the  data 
system(s). 

(3)  To  make  sure  the  Data  Project  Plan  (DPP),  which  is  the  plan  for  the 
management,  development,  and  implementation  of  the  system(s),  is  clear,  complete, 
correct,  and  consistent  with  the  direction  in  the  Data  Project  Directive  (DPD). 

(*»)  To  form  the  basis  for  establishing  the  Functional  Baseline. 

b.  System  Design  Review  (SDR). 

(1)  To  make  sure  the  functions  documented  in  the  System  /Subsystem  Specifi¬ 
cation  (SS),  which  are  needed  to  satisfy  each  requirement  documented  in  the  FD,  have 
been  assigned  (allocated)  to  specific  programs  (components)  in  the  data  system(s). 

(2)  To  make  sure  the  functions  to  be  carried  out  by  each  program  in  the  data 
system(s)  will  satisfy  the  requirements  of  the  functional  OPR  documented  in  the  FD. 

(3)  To  make  sure  all  manual  (human),  automatic  data  processing  equipment 
(ADPE),  and  data  system  (software)  interfaces  have  been  named,  defined,  and 
negotiated. 

(4)  To  form  the  basis  for  establishing  the  Allocated  Baseline. 

c.  Preliminary  Design  Review  (PDR). 

(1)  To  make  sure  the  general  (macro)  technical  design  of  each  program 
(component)  in  the  data  system(s)  is  acceptable  and  will  satisfy  the  requirements  of  the 
functional  OPR  documented  in  the  FD. 

(2)  To  make  sure  the  programs  that  make  up  the  data  system(s)  will  satisfy  the 
requirements  ol  the  functional  OPR  documented  in  the  FD. 


d.  Critical  Design  Review  (CDR). 
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(It  To  riidKe  sore  cite  detailed  technical  design  ol  each  program  in  the  data 
systein(s)  will  sutisly  its  intended  purpose  (satisty  tne  requirements  ol  the  lunctiona: 
OPK)  prior  to  programming  (coding). 

(2)  To  make  sure  all  support  documents  (Users  Manual  (UM),  Operations 
Manual  (OM),  Maintenance  Manual  (MM),  and  Test  and  Implementation  Plan  (PT))  are 
clear,  complete  (draft),  correct,  and  consistent  with  the  data(s)  to  be  programmed. 

e.  Product  Verification  Review  (PVR). 

(1)  To  make  sure  the  product  (Computer  Program  Configuration  Item  (CPC!)) 
that  has  been  programmed  and  tested  by  data  automation  satisfies  all  functional  and 
technical  requirements. 

(2)  To  make  sure  ail  associated  documentation  agrees  with  the  product  that 
has  been  programmed. 

(3)  To  make  sure  the  product  is  ready  for  functional  user  testing  (validation 
testing)  and  the  preparations  for  the  Test  Phase  have  been  completed. 

(4)  To  form  the  basis  for  establishing  the  Product  Baseline. 

f.  System  Validation  Review  (SVR). 

(1)  To  confirm  that  system  performance,  as  proven  during  validation  testing, 
meets  the  functional  OPR  (proponents)  requirements  documented  in  the  FD  and  DAR(s). 

(2)  To  make  sure  the  system  is  technically  acceptable  for  operation  in  a 
day-to-day  environment. 

(3)  To  make  sure  all  necessary  arrangements  have  been  made  for  implementa¬ 
tion  and  operational  support  of  the  system. 

Preparing  for  Reviews.  Preparation  is  the  single  most  important  factor  contributing  to 
the  success  of  any  Afk  300-15  review.  It*s  the  responsibility  of  the  PM/DPM  to  see  that 
preparations  for  a  review  are  complete  before  committing  external  project  resources  to 
it.  A  description  of  the  activities  usually  associated  with  preparing  for  any  review  are 
named  and  described  in  the  following  subparagraphs.  It  must  be  emphasized,  these  are 
activities  usually  associated  with  preparing  for  any  review.  Since  each  project  varies  in 
scope,  complexity,  and  direction  received,  it  would  be  impossible  to  name  all  possible 
activities  associated  with  each  review  for  all  possible  projects.  Realizing  this,  each 
PM/DPM  must  fully  understand  the  specific  purpose  of  each  review,  must  comply  with 
direction  and  directives  in  tailoring  the  review  to  his/her  project,  and  must  make  sure  all 
prerequisite  activities  are  completed  prior  to  a  review.  The  following  are  the  usual 
activities. 

a.  Identifying  Items.  The  PM/DPM  must  first  name  the  items  that  will  be  assessed 
and  verified  for  completeness  prior  to  and/or  during  a  particular  review.  To  make  this 
determination,  the  PM/DPM  should  rely  on  AFR  300-15,  Chapter  3,  the  projects  Data 
Project  Directive  (DPD)  and  Data  Project  Plan  (DPP),  as  well  as  Volume  1,  Part  A  of  this 
handbook.  A  list  of  the  reviews  and  the  items  usually  reviewed  follow: 

(1)  SRP  -  DAR(s),  DPD(s),  DPP,  FD,  and  ail  amendments. 
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2 (OR  -  bame  as  aDove  plus  me  S5  (Sections  1,  I,  /  i).  P '  u!  prepared!, 

configuration  management  recoras.  anc  previous  review  minutes. 

i3l  Same  as  above  plus  trie  completed  SS  and  t  araf:  ol  the  PT. 

>4)  CDR  *  Same  as  above  except  fina.  PT  (less  Section  2),  PSs,  Di  (if  preparedi 
anc  a  d^ctt  of  tne  MM,  OM.  and  UM. 

l5)  PVR  -  Same  as  aDove  except  final  PT,  MM,  OM,  and  UM. 

(6)  SVR  -  Same  as  aoove  plus  the  RT. 

b.  Determining  Acceptability.  The  PM/DPM  is  responsible  ior  making  sure  the 

item(s)  to  be  reviewec  are  prepared  in  accordance  with  governing  directives  before 
sending  it  to  people  1  or  review.  The  PM/DPM  also  has  an  obligation  to  make  sure,  to  the 
best  of  his/her  ability,  the  itemts)  is  a  quality  product  tnat  will  serve  its  intended 
purpose.  To  make  sure  a  quality  item  will  be  delivered  the  PM/DPM  should,  at  key 

points,  monitor  the  completion  of  tasks  associated  with  the  preparation  of  the  item. 

This  can  be  done  by  scheduling  walk-throughs,  audits,  briefings,  interim  reports,  etc.  As 
an  example,  the  PM/DPM  should  review  the  results  of  the  functional  systems  analysis 
tasks,  before  Sections  2  and  3  of  the  FD  are  started /completed.  By  scheduling  and 
conducting  such  internal  reviews  the  PM/DPM  can  preclude  the  delivery  of  items  that 
have  not  oeen  prepared  according  to  direction  and  do  not  serve  the  purpose  for  which 
they  were  intended. 

c.  Identifying  Participants.  The  identification  of  people  to  review  items  and/or 
attend  reviews  is  critical  to  the  success  of  a  project.  The  PM/DPM,  working  with 
functional  and  development  OPRs/OCRs,  should  find  and  gain  commitment  of  function¬ 
ally /technically  qualified  people  to  review  each  item  and/or  attend  each  review.  The 
PM/DPM  should  look  on  those  selected  as  constructive  critics  anc  not  as  adversaries. 
The  following  subparagraphs  give  more  specific  information  to  assist  the  PM/DPM  in  the 
selection  of  participants: 

(1)  Those  in  attendance  should  be  responsible  for  (PM/DPM,  OPR/OCR),  or  be 
able  to  make  a  material  contribution  (based  on  purpose  of  the  review)  to  the  success  of 
the  project. 

(2)  It’s  not  necessary  for  representatives  of  all  functional  user  organizations 
(as  opposed  to  functional  OPR/OCR),  interfacing  data  automation  organizations  (as 
opposed  to  the  development  OPR/OCR),  or  representatives  of  other  interfacing  organi¬ 
zations  (SAC,  AFDSDC,  ArAFC,  etc.)  to  be  in  attendance.  However,  if  needed,  the 
PM/DPM  or  functional/development  OPR/OCR  can  ask  representatives  of  user /data 
automation/interfacing  organizations  to  review  documentation  and  supply  Design  Prob¬ 
lems  Reports  (DPRs)Zcoordination.  The  PM/DPM  or  functional  /development  OPR  may 
also  ask  representatives  from  these  same  organizations  to  act  as  advisors  at  reviews 
when  their  attendance  will  materially  contribute  to  the  success  of  the  project. 

(3)  When  the  project  manager  expects  controversy  or  plans  changes  in  the 
project  which  will  need  review  and  approval  (RCB,  PMR,  CMR,  HQ  AF.  etc.), 
representatives  from  organizations  responsible  for  giving  or  coordinating  on  project 
direction  (HQ  AF/LE/AC,  HQ  AFLC/AC/SC/XR,  AFDSEC,  AF  A  A,  etc.)  should  be  in 
attendance.  The  knowledge  gams  by  these  representatives  will  be  instrumental  in 
enabling  them  to  help  in  resolving  controversy  and  in  forming  an  organizational  position 
on  project  direction  (continue,  redirect,  end). 
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(4)  The  puroose  of  each  review  is  different,  so  oarticipation  can  be  different. 
As  an  example,  the  functional  OPR  may  plan  for  selected  functional  users  to  attend  th^ 
SRR,  but  not  the  SDR.  Likewise,  the  development  OPR  may  plan  for  ail  lead 

Sogrammers  to  attend  the  CDR,  but  not  the  SRR.  Further,  the  data  system  support 
ISS)  manager  may  be  in  attendance  at  the  SRR  to  answer  questions  about  project 
direction  in  the  DPD,  but  may  not  attend  the  CDR  because  it  is  not  a  ’’baseline"  review. 

(5)  Each  project  manager,  when  preparing  the  DPP  for  his/her  project,  should 
describe  in  Section  VI  of  the  Configuration  Management  Plan  (CMP),  which  is  an 
appendix  to  the  DPP,  plans  for  conducting  project  reviews  and  audits.  These  plans,  in 
addition  to  identifying  the  reviews/audits  to  be  conducted,  will  name  who  the  reviewing 
authorities  will  be.  When  the  DPP  is  coordinated,  organizations  involved  in  the  project 
m«y  suggest  (with  explanations)  adding/dropping  review  participants  as  proper. 

d.  Planning  Format.  The  format  for  a  review  meeting  can  vary  depending  on  the 
project,  the  review  being  conducted,  and  the  preferences  of  the  PM/DPM.  The  PM/DPM 
should  always  develop  and  document,  in  the  proposed  agenda,  the  format  of  the  meeting. 
In  developing  the  format  the  PM/DPM  should  always  remember  the  specific  purpose  of  a 
review  and  see  the  purpose  is  served.  A  general  outline  for  reviews  which  has  proven 
successful  and  keeps  the  PM/DPM  in  control  follows. 

0)  Opening  Remarks  -  Given  by  PM/DPM  to  explain  purpose,  format,  proce¬ 
dures,  and  administrative  matters. 

(2)  Introductions  -  Given  by  PM/DPM  to  introduce  all/key  participants  and 
their  roles,  relationship  to  project,  and  responsibilities. 

(3)  Project  Background 

(a)  Functional  -  Given  by  functional  OPR/OCR(s)  to  explain  what  is  to  be 

accomplished. 

(b)  Data  Automation  -  Given  by  development  OPU/OCRfs)  to  explain  how 
data  automation  will  be  used  to  satisfy  the  requirements  of  the  C 'PR/OCR (s). 

(c)  Project  Management  -  Given  by  PM/DPM  to  explain  history  of  project 

to  date. 

M  Items  to  be  Reviewed  -  Given  by  PM/DPM  to  name  items  to  be  reviewed, 
their  purposes,  and  their  relationships. 

(3)  Review  of  Items  *  Led  by  PM/DPM  to  assess  and  verify  the  completion  of 
tasks  related  to  milestones. 

(6)  Summary  of  Open  DPR(s)  -  Identified  by  PM/DPM  to  inform  participants  of 
actions  to  be  completed  before  review  is  completed. 

(7)  Plans  for  Completing  Items  -  Given  by  PM/DPM  to  inform  participants  of 
what  actions  will  be  taken  when  to  update/"baseline"  review  Items. 

(8)  Plans  for  Coordination  -  Given  by  PM/DPM  to  inform  participants  of  what 
coordination  is  needed  and  when  it  will  be  carried  out  to  complete  the  review. 
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(9)  Future  Plans  -  Giver  b\  PM/DPM  to  inlorm  pa—.i-.ipants  of  near  tern 
project  tasks. 

(10)  Closing  -  Given  b>  PM/DPM. 

Preparing  Agenda.  The  PM/DPM  is  responsible  1  or  preparing  an  agenda  which 
outlines  the  format  for  each  review,  lr.  additior  to  naming  the  major  subject  and 
primary  participants,  the  agenda  should  include  location's),  date(s),  start -stop  times, 
break/meal  times  and  other  pertinent  planning  information.  The  final  agenoa  should 
reflect  the  results  of  a  coordinated  planning  effort  which  most  effectively  makes  use  of 
Air  Force  time  and  funds.  (Reference  Atch  2  for  sample  agenda) 

f.  Sending  Material.  The  PM/DPM  is  responsible  for  assembling  and  sending,  with 
a  letter  of  transmittal,  the  proposed  agenda  and  material  to  be  reviewed.  The  letter 
should  be  addressed  to  all  participating  people/organizations.  It  should  identify  what  is 
being  transmitted  and  explain  what  is  expected  of  participants  prior  to  attending  the 
review.  As  an  example  the  PM/DPM  could  ask  the  agenda  be  reviewed  and  any  proposed 
changes  called  in  by  a  certain  date.  The  PM/DPM  could  also  ask  all  DPRs  be  sent  no 
later  than  5  workdays  prior  to  the  review.  The  PM/DPM  should  consider  the  amount  of 
material  to  be  reviewed  and  allow  adequate  time  for  a  thorough  review  of  the  material 
by  the  participants  and  enough  time  for  PM/DPM  review  of  the  DPRs. 

g.  Reviewing  Design  Problem  Reports.  The  PM/DPM  should  receive  all  DPRs, 
except  those  prepared  at  the  review,  prior  to  the  review  meeting.  Each  DPR  should  be 
reviewed  and  a  coordinated  position  developed  with  the  OPR /OCR  (s>  prior  to  the  review  . 
Those  DPRs  needing  work  to  resolve  that  will  extend  beyond  the  dates  of  the  5RR  should 
have  an  estimated  completion  (suspense)  date.  The  DPRs  should  be  assigned  control 
numbers  and  entered  into  the  DPR  Log.  The  PM/DPM  should  determine,  based  on  those 
DPRs  received,  how  best  to  address/present  them  at  the  review.  As  an  example,  all 
administrative  DPRs  (editorial  discrepancies)  could  be  aggregated  together  and 
addressed  all  together  by  deliverable  rather  than  individually.  The  PM/DPM  may  decide 
to  modify  the  review  agenda,  based  on  the  volume,  content,  and  validity  of  the  DPRs 
received. 


h.  Planning  Support.  The  PM/DPM  should  make  sure  all  administrative  support 
details  are  taken  care  of  prior  to  a  review.  These  details  include,  as  an  example: 

(1)  Secretarial  support. 

(2)  Lodging  and  transportation  for  participants. 

(3)  Responsibilities  of  project  office  people  prior/during  SRR. 

(4)  Supplies. 

(5)  Room(s)  for  meeting. 

Conducting  Review  Meetings.  Next  to  adequately  preparing  for  a  review,  properly 
conducting  the  review  meeting  is  the  most  important  factor  contributing  to  its  success. 
AFR  300-15  places  the  responsibility  for  chairing  all  reviews  on  the  PM  or  his/her 
designee.  The  chairperson,  for  the  purpose  of  this  handbook,  is  considered  to  be  the 
PM/DPM,  and  so  it  is  his/her  responsibility  to  conduct  all  review  meetings.  To  chair  a 
review,  the  PM/DPM  must  understand  the  purpose  of  it  and  must  plan  to  see  its  purpose 
is  satisfied.  Said  another  way,  "before  leading,  you  must  know  where  von  are  going  and 
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have  deemed  on  how  best  to  get  there.”  The  PM/DPM  can  best  satisfy  the  purpose  of  a 
review  by  adhering  to  the  "coordinated"  agenda  he/she  should  have  prepared.  The 
following  subparagraphs  give  an  example  of  opening  a  review,  conducting  a  review  of 
selected  items,  and  closing  the  review.  These  subparagraphs  were  prepared  based  on  the 
format  in  paragraph  d  above  which  is  assumed  to  have  been  incorporated  into  the  agenda: 

a.  Opening  a  Meeting.  The  PM/DPM  opens  the  review  by  introducing  himself/her¬ 
self.  This  introduction  is  followed  with  an  explanation  of  the  purpose  of  the  review. 
This  explanation  should  be  more  than  quoting  AFR  300-  15.  As  an  example;  "The  purpose 
of  this  SRR  is  to  review,  complete,  approve,  and  place  under  change  control  the  OPRs 
and  OCRs  ADS  requirements  defined  in  the  FD.  I  want  the  considered  opinions  of  those 
present  as  to  whether  the  stated  requirements  are  correct,  complete,  testable,  and 
communicate  to  the  end  user  and  development  OPR/OCR(s)  a  clear  statement  of  the 
operational  capability  to  be  developed.  I  also  want  to  review  certain  near-term  plans  so 
everyone  is  familiar  with  their  respective  responsibilities  and  tasks."  The  PM/DPM  next 
gives  an  explanation  of  how  he/she  intends  to  satisfy  the  stated  purpose  of  the  review. 
As  an  example,  ”1  will  first  review  the  FD,  section  by  section.  While  reviewing  a  section, 
I  will  discuss  each  DPR  against  that  section  and  its  proposed  disposition."  The  PM/DPM 
would  then  give  an  explanation  of  the  procedural  and  administrative  details.  As  an 
example,  "If  there  is  a  need  to  prepare  DPRs  during  this  review,  1  will  identify  and 
discuss  the  need  and  if  valid  the  DPR  form  will  be  prepared  by  the  originator,  assigned 
for  action,  and  given  a  control  number." 

Following  these  opening  remarks,  the  PM/DPM  should  introduce  all  participants  or  key 
individuals,  such  as  the  functional  OPR/OCR(s),  development  OFR/OCR(s)  and  people 
representing  other  Commands  or  services.  As  the  individuals  are  being  introduced,  their 
relationship  and  responsibility  to  the  project  should  be  explained.  As  an  example,  "Mr  A 
from  the  Strategic  Air  Command  is  responsible  for  acquiring  B  and  developing  C  and 
interfacing  it  with  the  D  part  of  this  project." 

Following  his/her  introduction  the  functional  OPR/OCR(s)  should  be  asked  to  give  an 
explanation  of  what  the  LMS  requirements  and  benefits  are  in  general  terms.  Such  as: 
"This  requirement  for  ADP  support  is  necessary  to  enable  the  XY2  organization 

to  -  -  -  If  successful  it  will  save  $X  and  decrease  response  time  to - This 

functional  background  explanation  should  be  followed  by  an  explanation  by  the  develop¬ 
ment  of  how  data  automation  will  satisfy  the  requirements  in  the  FD.  For  example:  "the 
requirements  in  the  FD  do  not  need  any  special  equipment  or  software,  so  we  will  use  the 
XYZ  as  configured  to  satisfy  all  requirements."  The  ADP  background  explanation  should 
then  be  followed  by  a  brief  background  of  the  project  by  the  PM/DPM.  As  an  example, 
"The  DAR  was  approved  in  March  and  the  DPD  was  received  in  April.  The  project  office 
was  formed  in  April  and  the  DPP  completed  in  June.  People  from  the  ADP  organization 
and  the  functional  OPR/OCR(s)  organization(s)  completed  their  systems  analysis  in 
August.  Subsequently,  the  draft  FD  was  completed  and  forwarded  to  each  of  you  in 
December."  The  PM/DPM  would  next  name  the  items  to  be  reviewed  and  tneir 
relationship  to  each  other.  He/she  might  say,  "The  DPP  will  be  reviewed  in  context  with 
the  direction  in  the  DPD.  We  received  only  I  minor  DPR  against  the  DPP."  Following 
the  agenda,  which  should  orient  all  participants  to  the  project  and  give  then-  an 
understanding  of  what  will  be  done  during  the  review,  the  PM/DPM  begins  the  review. 

Conducting  a  Review.  Following  the  opening  of  the  meeting,  the  PM/DPM  begins  the 
review  o {  the  first  item.  As  mentioned  before,  the  PM/DPM  must  tailor  the  review  to 
his/her  project.  If  the  project  is  very  large  and  complex  and  involves  large  numbers  of 
people  in  the  review  meeting,  the  review  should  be  more  structured  so  the  PM/DPM  can 
remain  in  control.  If  the  opposite  is  true,  the  meeting  can  be  more  informal.  In  either 


case  the  PM/DPM  mus:  realize  tna:  he/sne  is  in  charge  and  is  resDonsible  ior  making 
sure  the  purpose  ol  having  the  review  is  met.  The  following  paragraphs  address  each  of 
the  six  AFR  300-15  reviews  and  gives  examples  ol  how  a  PM/DPM  could  conduct  each 
review  meeting.  Within  the  subparagraphs  are  points  the  PM/DPM  could  choose  to  b-ing 
up  concerning  the  purpose  ol  parts  ol  the  items  being  reviewed  and  relationships  ol  parts 
within  an  item  and  between  items.  Futher  explanations  on  the  items  1DOD7935.1-S 
aocuments)  to  be  reviewed  will  be  contained  in  a  later  edition  ol  this  volume  ol  the 
handbook. 

a.  Svstem  Requirements  Review'  (SR.R).  The  SRR,  for  purposes  of  this  handbook, 
begins  with  the  Pm/DPM  identifying  the  D*iR,  DPD,  DPP,  and  FD  as  the  items  to  be 
reviewed.  He/she  also  points  out  that  DPRs  will  be  addressed,  when  applicable,  to  the 
part  of  the  item  being  reviewed. 

The  DAR  and  all  amendments  to  it  should  be  briefly  summarized  by  the  PM/DPM. 
He/she  might  state,  "each  of  vou  have  a  copy  of  the  original  DAR.  There  has  been  1 
amendment  to  it  which  added  the  requirement  to  do  A.  This  additive  requirement  added 
$B  in  cost,  C  man-hours  and  D  additional  days  to  E  task.  The  benefit  is  estimated  to  be 
$F  per  year.  These  changes  did  not  exceed  the  15%/ 120  day  thresholds  where  a 
management  review'  is  required.  The  certified  Economic  Analysis  (EA)  is  still  valid  and 
represents  the  anticipated  costs  for  this  project.  The  last  document  to  be  reviewed  will 
be  the  FD  and  all  requirements  therein  emenate  from  this  DAR  and  1  amendment." 
After  the  discussions,  if  any,  the  DPD  is  reviewed. 

The  PM/DPM  should  briefly  summarize  the  direction  in  the  DPD  and  any  amendment 
which  is  pertinent  to  the  review.  This  could  include  direction  such  as:  w>hat  items  must 
be  completed  in  the  Conceptual  Phase,  who  is  responsible  for  producing  a  particular 
item,  what  reviews  must  be  held,  what  level  of  approval  is  required  if  thresholds  are 
breached,  etc.  After  reviewing  the  relevent  direction  in  the  DPD,  the  DPP  is  reviewed, 
In  some  cases  the  PM/DPM  may  wish  to  summarize  the  DPD  first. 

The  PM/DPM  should  discuss  with  review  participants  the  major  milestones  shown  in  the 
DPP.  Following  this  brief  review  the  near  term  tasks  and  their  OPR/OCS(s)  should  be 
reviewed.  If  there  are  schedule,  cost,  or  resource  changes  expected  they  should  be 
addressed.  The  PM/DPM  should  also  explain  how  changes  to  the  Functional  Baseline 
affecting  cost,  schedule,  or  performance  must  be  coordinated  and  approved.  The 
PM/DPM  should  also  explain  his/her  plans  to  add  other  appendices  to  the  DPP  and  the 
procedures  for  making  and  coordinating  changes  to  those  already  prepared.  Sometimes 
the  review  of  the  DPP,  because  of  already  planned  changes,  would  be  better  served  bv 
reviewing  it  last.  If  so,  the  agenda  could  be  prepared  showing  the  DPP  w-ill  be  reviewed 
following  Section  5  of  the  FD. 

The  next  and  most  critical  document  to  be  reviewed  is  the  FD.  The  FD  should  be 
thoroughly  reviewed  section-by-section  to  make  sure  all  attendees  understand  and  agree 
with  the  contents.  The  PM/DPM  should  point  out  that  the  acceptance  of  the  FD  as  a 
document  which  adequately  defines  the  functional  baseline  is  critical  to  the  success  of 
the  entire  project.  Without  a  defined  and  controlled  baseline  to  work  from,  the 
accomplishment  of  the  ADS  design,  the  preparation  of  the  User  Manual  (UM)  and  the 
preparation  of  the  Test  Plan  (PT)  become  an  impossible  task.  The  PM/DPM  should 
recognize  and  emphasize  the  importance  of  certain  paragraphs  within  sections  of  the  FD. 
As  an  example,  the  functional  oPR/OCR(s)  should  make  sure  an  adequate  analysis  of  the 
impact  of  the  changes  on  the  functional  user  organization  has  been  done  and  the  results 
clearly  documented  in  paragraph  2.6.1.  Likewise,  the  ADP  OPR/OCR(s)  should  carefully 
analyze  the  timing  requirements  (paragraph  3.1.2)  of  each  requirement  (paragraph  3.1) 
and  make  sure  the  proposed  environment  described  in  Section  &  is  capable  of  meeting 
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these  requirements.  If  not,  the  known  constraints  or  limitations  should  be  identified  in 
paragraph  2.7.  The  PM/DPM  should  also  recognize  and  point  out  that  particular 
attention  must  be  paid  to  the  contents  of  Section  3,  since  they  are  the  basis  lor  the 
contents  of  follow-or  documents.  As  an  example,  the  functions  identified  in  paragraph 
3.2  are  further  analyzed  and  detailed  in  the  System/Subsystem  Specification  (SS), 
paragraph  2.2.  The  functions  in  the  SS  are  allocated  to  programs  and  then  fuither 
analyzed  and  detailed  in  the  Program  Specifications,  paragraph  2.2.  Paragraphs  4.1.1 
and  4.1.2  of  the  PT  are  likewise  dependent  on  specific  and  testable  requirements  and 
functions  being  identified  in  paragraphs  3.1  and  3.2  of  the  FD.  The  PM/DPM  should  use 
visual  aids  such  as  charts,  diagrams,  etc.,  to  assist  in  the  review  of  the  FD  when  feasible. 

After  completing  the  review  of  each  section  of  the  FD,  and  of  each  DPR  associated  to 
each  section,  the  review  itself  is  considered  complete.  The  PM/DPM  now  closes  the 
review  meeting. 

b.  System  Design  Review  (SDR).  As  with  the  SRR,  the  SDR  begins  with  the 
PM/DPM  naming  the  first  item  for  review.  Experience  has  shown  the  PM/DPM  should 
first  attempt  to  familiarize  all  participants  with  the  current  Functional  Baseline  (FBL). 
This  can  be  carried  out  by  the  PM/DPM  briefly  going  over  the  SRR  minutes  and  then 
identifying  and  reviewing  any  changes  to  the  FBL  since  the  SRR  was  concluded.  This 
would  include  new  DARs/DPDs,  DAR/DPD  amendments,  Class  1  Baseline  Change 
Requests  (BCRs)  against  the  FBL  (FD),  and  changes  in  costs  and  benefits.  This  brief 
review  serves  to  give  all  participants  a  common  understanding  of  the  FBL. 

Before  beginning  the  review  of  the  next  item,  the  System  Subsystem  Specification  (SS), 
the  PM/DPM  should  explain  whether  the  OPR/OCR(s)  requirements  will  be  satisfied  by 
one  or  more  Computer  Program  Configuration  Items  (CPCIs).  As  pointed  out  in 
AFR  300-15,  if  there  are  multiple  CPCIs  each  is  required  to  successfully  complete  a 
SDR.  Basically  what  this  means  to  the  PM/DPM  is:  if  the  system  to  be  designed  will 
consist  of  and  be  managed  as  a  single  entity  (CPCI)--then  a  System  Specification  (SS' 
will  define  the  design  and  will  be  reviewed  at  a  SDR.  If  the  system  will  consist  of 
several  manageable  entitites  or  the  development  and/or  implementation  will  be 
phased — then  the  design  of  each  CPCI  will  be  defined  in  separate  Subsystem  Specifica¬ 
tions  each  having  a  separate  SDR.  DOD  7935.1-5  states,  "If  individual  Subsystem 
Specifications  are  prepared,  they  may  at  some  point  be  bound  together  to  form  a  System 
Specification  or  a  separate  System  Specification  may  be  written."  When  multiple  SSs  are 
being  prepared,  the  PM/DPM  must  so  advise  the  reviewers  so  they  are  aware  of  what 
requirements  in  the  FD  will  be  satisfied  by  a  CPCI  and  how  each  CPCI  relates  to  the 
others  in  the  overall  system.  For  the  purpose  of  this  handbook  a  single  CPCI  will  be 
assumed. 

After  attempting  to  set  up  this  common  point  of  reference  for  the  FBL,  the  SS  should  be 
reviewed  against  it.  That  is,  the  55  should  be  reviewed  against  the  Functional  Baseline 
defined  by  the  FD.  The  SS  should  be  reviewed  section  by  section  and  all  DPRs  against  a 
section  addressed  while  that  section  is  being  reviewed.  The  PM/DPM  should  ask  the 
development  OPR  to  give  an  overview  of  the  proposed  automated  data  system  IADS). 
This  could  be  done  from  the  charts  required  to  satisfy  paragraph  2.1  of  the  SS.  The 
PM/DPM  should  next  point  out  to  the  participants  that  the  System/Subsystems  Functions 
documented  in  paragraph  2.2  maintain  a  direct  relationship  to  both  the  requirements  and 
functions  in  the  FD,  paragraphs  3.1  and  3.2  respectively.  Again,  this  could  be  illustrated 
by  a  chart  showing  the  direct  relationship  of  SS  functions  to  FD  functions  and 
requirements  (reference  attachment  3).  The  PM/DPM  must  also,  as  required  by  AFLC 
Supplement  I  to  DOD  7935. 1-S,  name  what  functions  have  been  allocated  to  what 
programs  (components)  and  where  this  allocation  is  documented.  This  relationship  '-ould 


also  be  illustrated  d>  matrix  chart  (reference  attachment  9).  ‘rn*  PM/DPM  snouid  nex: 
ask  the  development  OPR  to  explain  the  contents  ol  Section  3  vnict-  defines  the  ADP 
environment  that  will  supoo-t  the  OPR  /OCR  (s'  requirements. 

Following  the  review  ol  Sections  1  through  3  of  the  S5  and  all  associated  DPRs.  the 
PM/DPM  snouid  name  the  next  item  for  review.  A  similar  review,  section  bv  section 
with  DPRs,  should  be  carriec  out  against  this  item  and  all  others,  if  feasible. 

After  reviewing  the  final  item,  the  PM/DPM  should  present  the  results  of  am 
simulations  of  the  proposed  design  in  the  SS  or  any  other  evidence  not  previous! v 
presented.  The  PM/DPM  should  entertain  any  new  DPRs,  discussions  or  questions 
concerning  the  SDR  items.  After  discussions,  if  any,  the  PM/DPM  should  advise  the 
participants  the  review  of  the  items  is  completed  and  close  the  SDR  meeting. 

c.  Preliminary  Design  Review  (PDR).  Before  discussing  the  meeting,  it  is 
important  to  explain  several  inconsistencies  in  AFR  300-15  regarding  PDRs.  Following 
this  discussion  will  be  an  explanation  of  the  inconsistencies  and  of  what  AFLC/SC 
recommends  to  eliminate  these  inconsistencies. 

AFR  300-15,  figure  1-2  identifies  the  SS  and  Subsystem  Specifications  (55s)  as  being 
separate  items  that  are  reviewed  at  the  SDR  and  PDR  respectively.  However, 
DOD  7935.1-S,  paragraph  2.9.3  clearly  points  out  that  either  a  SS  or  SSs  will  be 
prepared,  not  both.  Further,  AFR  300-15,  figure  2-1,  doesn't  identify  the  SSs  as  being 
separate  and  distinct  items.  It  show's  a  SS  or  SSs  being  placed  under  change  control  after 
a  successful  SDR.  This  position  is  consistent  with  the  statement  in  AFR  300-15, 
paragraph  l-9a,  but  is  contradicted  by  paragraph  3-7a(2).  These  are  but  a  few  of  the 
inconsistencies  surrounding  the  PDR  and  SS/SSs  in  AFR  300-15.  To  eliminate  debate  and 
meet  the  intent  of  DOD  7933.1  -S  and  AFR  300-15,  and  also  stay  consistent  w»ith  the  way 
ADSs  evolve,  SC  takes  the  following  position.  If  there  are  multiple  CPCls  in  a  ADS. 
each  of  their  designs  will  be  documented  in  separate  Subsystem  Specifications  and  be 
subjected  to  SDRs  and  PDRs.  Conversely,  if  there  is  only  one  CPC1,  its  design  will  be 
documented  in  a  System  Specification  and  reviewed  at  a  SDR  and  PDR.  If  the  PM/DPM 
wants  the  design  of  the  ADS  system,  which  is  defined  in  Sections  2  and  3  of  the  SS/SSs 
assessed  and  verified,  this  car  be  done  at  a  SDR,  without  the  finite  details  for  each 
program  being  documented  in  Section  9.  In  large  systems  this  could  save  considerable 
rework  in  SectiornTTTthe’design  of  the  ADS  is  found  to  be  faulty.  This  is  not  to  say  the 
details  in  Section  9  are  not  available,  they  just  are  not  available  in  finished  form  as 
Section  9  requires.  Once  having  successfully  completed  a  SDR,  and  established  an 
Allocated  Baseline  (ABU,  the  agreed  on  controlled  design  (macro!  of  the  ADS  can  now  be 
defined  (documented)  at  the  program  level  in  Section  9  of  the  SS/SSs.  This  level  of  detail, 
which  logically  follows  the  design  of  the  ADS  system  or  subsystem,  can  now  be  assessed 
and  verified  at  a  PDR.  Following  the  PDR,  each  program  documented  in  Section  9  will 
be  designed  at  the  lowest  level  and  documented  in  a  Program  Specification.  If  the 
PM./DPM  decides  the  ADS  design  has  little  chance  of  being  found  faulty,  he/she  can 
conduct  a  combined  SDR/PDR  at  which  time  a  complete  (Sections  1  through  9) 
System/Subsystem  Specification  is  reviewed.  In  summary  AFLC/SC  believes  the  logical 
evolution  of  a  ADS  system  or  subsystem  is  documented  first  in  either  an  SS  or  in  SSs  no: 
both.  The  ADS  level  design  is  documented  in  Sections  2  and  3,  which  is  reviewed  at  the 
SDR,  followed  by  the  documentation  of  the  program  level  (macro)  design  in  Section  9. 
which  is  reviewed  at  the  PDR. 

Following  the  opening  of  the  review,  the  PM/DPM  reviews  the  SDR  minutes  and 
summarizes  previous  changes/additions  to  the  FBL  and  Allocated  Baseline  (ABL).  After 
any  discussions,  the  review  of  Section  9  of  the  SS  is  begun.  Section  9  should  be  reviewed 
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in  (('i  irvi  c»  : in  the  arid  KBL  That  is,  the  participant1  sl.mikt  assess  and  ve'ify 

whetner  the  tna<.  ro  !eve.  design  of  eo'n  program  will  enable  the  functions  carried  oul  bv 
it  to  meet  tht  OPR-'«'CRs  requirements.  Farther,  each  prograni  must  he  designed 
according  to  applicable  standards  and  be  able  to  accomplish  the  functions  allocated 
within  the  technical  limitations  of  the  environment  defined  in  Section  1.  Dust  as  the 
name  infers,  it  is  a  preliminary  design  review  of  each  component  (program)  in  the  ADS 
system  or  subsystem. 

The  last  item  to  review  is  a  draft  of  the  Test  and  Implementation  Plan  (PT).  Draft  in 
this  context  means  a  PT  in  reviewable  form,  excluding  the  results  of  development  testing 
required  in  Section  2.  Tne  draft  PT  is  reviewed  at  this  time  to  assess  and  verify  the  ADS 
and  programs  as  designed  are  capable  of  being  tested  to  demonstrate  the  OPR /OCR  (s) 
requirements  have  been  met.  If  not,  either  the  ADS  design,  program  design  or  the  tests 
must  be  redesigned.  The  PM/DPM  should  explain  the  purpose  of  the  PT,  as  explained  in 
Section  1  and  also  explain  that  the  tests  specified  must  relate  to  the  requirements  and 
functions  identified  in  the  FD. 

After  completing  the  review  of  the  SS,  PT,  and  any  associated  DPRs,  the  PM/DPM 
should  close  the  meeting. 

d.  Critical  Design  Review  (CDR).  The  PM/DPM  begins  the  review  by  identifying 
the  first  item  to  be  reviewed.  As  with  each  review,  the  PM/DPM  should  attempt  to 
bring  all  participants  up-to-date  as  to  the  FBL  and  ABL.  As  explained  previously,  this  is 
done  by  briefly  discussing  the  PDR  minutes  and  all  approved  changes  made  to  the  system 
since  the  last  review.  The  PM/DPM  will  find  the  configuration  management  (CM)  status 
accounting  records  can  be  useful  in  accomplishing  this  update. 

Following  the  update,  the  PM/DPM  names  the  Program  Specification  (PS)  that  will  be 
reviewed.  The  PM/DPM  may  decide,  in  the  case  of  large  systems  and 'or  when  no  DPRs 
have  been  received  against  a  PS,  that  all  PSs  will  not  be  reviewed  at  tie  meeting.  If  the 
PM/DPM  decides  all  PSs  will  not  be  reviewed,  he/she  should  explain  why  certain  ones 
will  and  others  will  not.  The  PSs  to  be  reviewed  should  be  reviewed  against  the  SS(s), 
wh.ch  names  the  functions  to  be  carried  out  by  each  program  and  its  inputs,  outputs, 
data  bases,  interfaces  (internal/external).  As  explained  by  DOD  7935.1-S,  pages  3-33, 
through  3-90,  the  definition  of  each  program  must  relate  to  specific  parts  of  the  SS(s) 
and/or  FD.  Each  PS  reviewed  must  be  assessed  and  verified  as  to  its  ability  to  meet  the 
OPR/OCR(s)  requirement,  to  its  conformance  to  standards,  and  to  its  technical  correct¬ 
ness.  When  the  PSs  are  approved,  they  will  be  the  basis  for  each  programmer  to  code 
from.  This  is  not  to  say  that  in  certain  cases  where  risk  is  low  or  techniques,  algorithms, 
etc.,  require  verification,  that  coding  has  not  already  begun  or  completed  for  selected 
programs.  If  this  has  been  done  the  PM/DPM  should  so  advise  and  use  the  results  to  aid 
in  the  review. 

When  the  review  of  the  proposed  technical  design  of  all  or  critical  programs  has  been 
completed,  the  system  support  documents  should  be  reviewed.  These  include  drafts  of 
the  Users  Manual  (UM),  Operations  Manual  (OM\  and  although  considered  by  AFR  300-  12 
to  be  an  internal  document,  the  Maintenance  Manual  (MM),  and  a  completed  Test  Plan 
(PT).  These  documents  should  be  reviewed  in  context  with  the  FD,  SS(s),  and  PS(s).  They 
must  be  reviewed  to  decide  if  they  are  correct  ond  consistent  given  the  ADS  design 
solution  specified  in  the  SS,  PS(s),  and  RD  and  DS  i  prepared.  The  ultimate  obiertive  of 
course  is  to  make  sure  the  ADS  to  be  coded  can  be  used  by  the  functional  user,  operated 
by  operations  personnel,  tested  by  an  independent  t?st  team,  and  maintained  bv  the  ADP 
maintenance  staff.  If  design  deficiencies  are  notec  during  this  review,  the  svstem/pro- 
gram  design  must  be  corrected  and  baseline  documents  changed  accordingly. 
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After  completing  the  reviev  o:  all  item*  tnt  PM'DDM  should  ciose  the  meeting. 

Product  Verficauor.  Review  (PVR).  The  PM/DPM  shoulc  open  the  PVR  witn  t 
brief  sumrT.a-v  of  tne  results  c:  the  preliminary  Physical  and  Functional  Configuration 
Auoits  and  the  result  of  development  testing.  This  should  be  followed  witf  a  discussion 
of  the  changes  to  tne  FBi.  and  ABL  that  have  taken  place  since  the  CDR. 

Following  these  discussions,  the  audit  report  is  reviewed  anc  the  item*  previous!) 
reviewec  at  the  CDR  are  reviewed  in  their  final  form.  Tne  PM/DPM  snouid  explain  to 
all  participants,  tnat  once  approved  the  support  documents  will  be  used  b\  all  people, 
functional  and  ADP.  during  the  Test  Phase.  Therefore,  each  and  every  document  should 
be  reviewed  as  if  it  were  ready  for  lull  operational  use.  The  PM/DPM  should  also  point 
out  that  subsequent  to  tneir  review  and  approval,  the  PS(s)  which  define  the  Product 
Baseline  will  be  placed  under  configuration  control  (Class  1/11  BCR). 

Alter  completing  the  review  and  addressing  ail  DPRs  the  meeting  is  closed. 

1.  System  Validation  Review  (SVR).  This  review,  which  is  conducted  at  the  end  of 
the  Test  Phase,  is  conducted  to  assess  and  verify  the  systems  acceptability  lor 
operational  use.  To  accomplish  this  and  terminate  the  development  project,  the 
PM/DPM  must  get  the  functional  OPR/OCR(s)  to  certify  the  system  that  has  been 
developed.  To  accomplish  this,  the  PM/DPM  should  review  the  Test  Analysis  Report 
(RT)  prepared  by  the  independent  test  director.  This  report  should  be  reviewed  tc  decide 
if  the  tests  conducted  provide  evidence  that  the  functional  OPR/OCRts)  requirements 
specified  in  the  FD  have  been  satisfied.  Additionally,  the  results  from  the  final  FCA  and 
PC  A  should  be  reviewed  to  decide  if  a  quality  product  has  been  developed  that  meets  all 
functional  requirements  and  technical  standards.  Following  these  reviews,  the  PM/DPM 
should  name  and  review  the  planned  disposition  of  all  outstanding  DPR(s),  System 
Problem  Reports  (SPRs)  (prepared  during  validation  testing),  and  BCRs. Following  their 
review,  the  PM/DPM  should  identify  to  the  representative  of  the  ADP  maintenance 
organization  all  of  the  items  that  will  be  turned  over  and  the  planned  turnover  date(s). 

After  the  above  has  been  accomplished  the  meeting  should  be  closed. 

g.  Final  Operational  Evaluation  (FOE).  This  evaluation  (review),  which  is  some¬ 
times  needed,  is  an  Operational  Phase  review  and  is  not  discussed  in  AFR  300- 1 5.  The 
description  of  this  review  is  described  in  AFR  300-12,  paragraph  4-5(6).  As  AFR  300-12 
explains,  the  FOE  "is  a  review  of  the  operational  data  system".  For  a  multi -site  system, 
this  review  will  take  place  when  the  system  is  operational  at  a  representative  set  ol 
planned  sites.  The  installation  at  the  remaining  sites  then  would  be  considered  to  be  the 
implementation  of  an  already  operational  system.” 

Since  there  is  nothing  more  definitive  concerning  this  review  or  the  organization 
responsible  for  its  accomplishment,  it  will  not  be  discussed  further  in  this  handbook.  The 
PM/DPM,  if  tasked  by  the  DPD  to  do  a  FOE,  should  ask  for  specific  instructions 
concerning  its  purpose,  items  to  be  reviewed,  reports  to  be  prepared,  etc. 

Closing  a  Meeting.  In  closing  a  review  meeting,  the  PM/DPM  should  try  to  wrap  up  all 
loose  ends,  this  gives  participants  an  understanding  of  what  actions  are  still  required  to 
successfully  conclude  a  review,  when  they  will  be  carried  out,  and  who  is  responsible  for 
their  accomplishment.  For  example,  the  PM/DPM  should  announce  when  meeting 
minutes  will  be  sent  and  name  all  DPRs  that  need  further  work  to  resolve.  For  each 
DPR  he/she  should  name  the  responsible  p«rs°n/organization  and  the  expected  comple¬ 
tion  date.  He /she  should  also  name  the  actions  still  needed  to  set  up  a  baseline  and  when 


they  will  be  completed,  e.g.,  complete  DPRs,  update  FD,  lorward  FD  to  key  participants 
with  AFLC  <*06  for  signature,  conduct  CCB  meeting.  Following  these  discussions,  the 
PM/DPM  should  reiterate  the  major  tasks  between  this  review  and  the  next  review. 
Accomplishing  this,  the  PM/DPM  should  then  thank  all  participants  for  assisting  h  m/her 
and  dismiss  the  participants.  After  the  meeting,  the  PM/DPM  must  now  complete  those 
actions  necessary  to  successfully  end  the  review. 

Concluding  Reviews.  After  a  review  meeting,  the  PM/DPM  must  take  certain  actions  to 
successfully  end  a  review  and  if  necessary  establish  a  baseline.  These  actions,  which  are 
specified  in  AFR  300-15  and  AFLC  Supplement  1,  are  discussed  in  the  following 
paragraph. 

The  first  concern  of  the  PM/DPM,  after  a  meeting,  is  to  make  sure  all  responsibilities 
are  assigned  and  understood.  Work  should  already  be  in  progress  toward  resolving  open 
DPRs,  and  the  meeting  minutes  should  be  nearing  completion,  if  secretarial  help  was 
available  during  the  meeting.  The  PM/DPM  should  have  a  letter  of  transmittal  ready  to 
transmit  the  updated  DOD  7935.1 -S  documents,  AFLC  <*06,  and  minutes  to  meeting 
participants  (Reference  Atch  5).  A  CCB  meeting  (to  recommend  acceptance  of  a 
document  for  baselining)  should  be  scheduled  and  prepared  for  as  well  as  a 
AFLCR  <*00-18  Program  Manager  Review  (PMR).  The  PM/DPM  should  also  make  sure 
the  required  CM  logs  are  up-to-date  and  procedures  have  been  set  up  to  maintain  the 
integrity  of  a  baseline. 

While  all  of  the  actions  to  conclude  a  review  seem  trivial,  it  must  be  remembered  there 
are  also  a  thousand  and  one  other  things  taking  place  concurrently,  relative  to  the 
project.  So  the  PM/DPM  should  recognize  and  be  ready  to  quickly  conclude  a  review  and 
have  a  defined  controllable  baseline  to  work  from.  This  is  not  to  say,  rush  to  completion. 
Rather,  it  is  saying  be  aware  of  all  of  the  activities  needed  to  end  a  successful  meeting 
and  make  sure  they  are  planned  for  and  are  carried  out  in  an  orderly  and  quick  manner. 


AUDITS 


Purpose  of  Audits.  Audits  o f  a  CPC1  are  conducted  at  two  points  in  its  development  to 
determine  whether  the  CPCJ  conforms  to  specifications  and  standards.  On  completion  of 
an  audit,  the  PM/DPM  will  receive  an  audit  report  citing  the  findings  and  recommenda¬ 
tions  of  the  audit  team.  A  more  detailed  explanation  ol  the  purpose  of  each  type  audit 
follows: 

a.  Functional  Configuration  Audit  (FCA).  This  audit  is  conducted  to  validate 
whether  a  CPCl's  actual  performance  complies  with  the  SS  and  meets  the  OPR/OCR(s) 
requirements  specified  in  the  FD. 

b.  Physical  Configuration  Audit  (PC A).  This  audit  is  conducted  to  validate 
whether  a  CPCI  agrees  with  its  technical  documentation,  conforms  to  quality  assurance 
policies  and  procedures,  and  adheres  to  applicable  standards. 

Preparing  for  Audits.  The  PM/DPM  must  get  ready  for,  as  noted  previously,  audits  at 
two  points  in  the  development  cycle  of  each  CPCI.  The  first  audit,  which  is  entitled, 
"Preliminary  Functional  and  Physical  Configuration  Audit,"  is  conducted  lead-time  away 
from  the  scheduled  PVR  for  a  CPCI.  The  second  audit  is  entitled,  "Final  Functional  and 
Physical  Configuration  Audit,"  and  is  conducted  lead-time  away  from  the  scheduled  SVR 
for  a  CPCI. 

To  prepare  for  an  audit,  the  PM/DPM  must  first  notify  the  audit  director,  named  in  the 
DPD,  of  the  particulars  concerning  the  CPCI  to  be  audited.  This  notification  is  then 
followed  with  the  delivery  of  the  items  to  be  audited  (reference  AFR  300-15,  Chapter  3, 
Section  C  for  contents  of  notification  anc  identification  of  the  items  to  be  audited). 

Conducting  Audits.  Conducting  audits  are  not  the  responsibility  of  the  PM/DPM. 
However,  the  PM/DPM  is  very  much  interested  in  their  results  because  they  are  a  key 
element  in  determining  whether  the  CPCI  is  ready  for  validation  testing  and  later  lor 
operational  use.  The  section  of  AFR  300- 1 5  referenced  in  the  preceding  paragraph 
shows  what  will  be  done  during  a  FCA  and  PC  A.  This  section  of  AFR  300- J  5  also  points 
out  the  difference  between  a  preliminary  and  final  configuration  audit. 

Concluding  Audits.  Both  audits  are  concluded  when  the  audit  director  prepares  minutes 
(report)  of  the  audit,  citing  the  findings  and  recommendations  of  the  audit  team.  These 
reports  are  given  to  the  PM/DPM  for  use  in  the  PVR  and  SVR.  The  PM/DPM  should 
review  these  reports  very  carefully  before  actually  conducting  a  PVR  or  SVR.  If  the 
findings  indicate  major  system  discrepancies  exist,  the  PM/DPM  must  then  determine  if 
the  PVR  and  SVR  should  be  delayed.  If  major  discrepancies  do  exist  and  the  PM/DPM 
decides  the  PVR/SVR  will  be  conducted  pt  ior  to  their  correction,  he/she  should  be  ready 
to  discuss  their  impact  on  validation  testing/operation  at  the  PVR/SVR/PMR. 


DOCUMENTS  NECESSARY  TO  BEGIN  THE  CONCEPTUAL  PHASE 

A.  Required  System  Capability  (RSC) 

1.  Prepared  in  accordance  with  (IAW)  AFLCR  400-5. 

2.  Prepared  by  organization  having  the  requirement  (OPR). 

3.  Validated  by  AFLC/XRB,  IAW  AFLCR  400-3,  attachment  2. 

4.  Prioritized  by  AFLC/XRB,  IAW  AFLCR  400-5,  attachment  3. 

5.  Approved  by  AFLC/XRB  or  AF/LE. 

6.  Entered  in  AFLC  Capabilities  Plan  (process,  priority,  etc.)  by  AFLC/XRB. 

7.  Entered  in  AFLC  LMS  Action  PLan  (objective,  schedule,  resources,  etc.)  by 
AFLC/SCC. 

B.  PROJECTED  ADP  REQUIREMENT  (PAR) 

1.  Prepared  IAW  AFLC/ACOR  Data  Call  Letter. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Presented  to  AFLC  Management  System  Panel  (MSP)  by  organization 
having  equipment. 

4.  Approved  and  prioritized  by  MSP. 

5.  Entered  into  AFLC  Major  Command  ADP  Plan  (MCAP)  by  AFLC/ACD. 

6.  If  manpower  is  required,  a  602  package  is  needed  at  the  same  time. 

C.  MANPOWER  CHANGE  REQUEST  (MCR) 

1.  AF  Form  602,  prepared  by  Servicing  Manpower  Evaluation  Team  (MET)  in 
cooperation  with  organization  having  requirement,  IAW  AFM  26-  l,  Chapter 
9. 

2.  Authenticated  by  HQ  AFLC/DPQ  and  sent  to  HQ  USAF/PRM. 

D.  PRELIMINARY  EVALUATION  NOTICE  (PEN) 

1.  Prepared  by  the  organization  having  requirement  IAW  AFLC  Supplement  1 
to  AFR  300-12. 

2.  Sent  to  AFLC/ACD  for  evaluation. 

3.  Evaluated  by  potential  data  automation  development  organization. 

4.  Returned  to  originating  organization  (via  ACD)  for  use  in  preparing  DAR. 
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E.  DATA  AUTOMATION  REQUIREMENT  (DAR) 


U  Prepared  IAW  APR  300-12,  attachment  l  and  AFLC  Supplement  I. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Validated  by  AFLC/XRB  IAW  AFLCR  400-5,  attachment  2. 

4.  Prioritized  by  AFLC/XRB  IAW  AFLCR  400-5,  attachment  3. 

5.  Scheduled  by  AFLC/ACD. 

6.  Approved  by  AFLC/ AC  or  AF/AC  it  LE  or  SAF/FM. 

7.  The  following  sequence  should  be  followed  FS,  EA,  OAR. 

F.  ECONOMIC  ANALYSIS  (EA) 

1.  Prepared  IAW  AFR  300-12,  Chapter  3,  Section  B  and  AFLC  Supplement  l, 
AFR  178-1  and  AFLC  Supplement  1. 

2.  Prepared  by  organization  having  the  requirement  (EA  needed  when 

estimated  development  man-years  are  five  or  more). 

3.  Certified  by  AFLC/ ACM  or  AF/ACM. 

4.  Included  (when  needed)  as  attachment  to  DAR,  IAW  AFR  300-12, 

attachment  l  and  attachments  8  thru  1 1. 

5.  Actual  costs  tracked  (when  needed)  IAW  AFR  300-12,  Chapter  3,  Section 
C,  attachments  12  thru  14. 

6.  Administrative  Personnel  assigned  to  a  project  office  must  be  costed  to  the 
project. 

G.  FEASIBILITY  STUDY  (FS) 

1.  Prepared  IAW  AFLC  Supplement  l,  attachment  3,  to  AFR  300-12. 

2.  Prepared  by  organization  having  the  requirement  (FS  needed  when 

estimated  man-years  are  10  or  more). 

3.  Included  (when  needed  as  attachment  to  DAR,  IAW  AFR  300-12. 

attachment  l. 

H.  PRIVACY  ACT  STATEMENT 

1.  Prepared  IAW  AFR  300-12;  Chapter  2,  paragraph  2-10. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Included  in  DAR  IAW  AFLC  Supplement  l,  attachment  6  to  ^FR  300-12. 
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(.  ADP  AND  TELECOMMUNICATIONS  REQUIREMENTS  CHECKLIST 

1.  Prepared  IAW  AFR  300-12,  Chapter  5,  6,  or  9. 

2.  Prepared  by  organization  having  the  requirement  (Checklist  prepared  when 
a  Delegation  of  Procurement  Authority  (DPA)  is  needed). 

3.  Certified  by  AFLC/AC,  AF/ACD,  or  SAF/FM. 

4.  Included  (when  needed)  as  attachment  to  DAR,  IAW  AFR  300-12, 
attachment  1  and  23. 

3.  All  documentation  supporting  the  checklist  must  be  kept  on  file. 

3.  SITE  PREPARATION  REQUIREMENTS 

1.  Prepared  IAW  AFR  300-12,  Chapter  2,  paragraph  2-11  and  AFLC  Sup¬ 
plement  1. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Included  in  DAR,  IAW  AFLC  Supplement  1,  attachment  6  to  AFR  300-  12. 

K.  AF/ATC  TRAINING  REQUIREMENTS  STATEMENT 

1.  Prepared  IAW  AFR  300-12,  Chapter  2,  paragraph  2-8. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Notification  of  training  requirement  (if  applicable)  to  Air  Training 
Command  (ATC). 

4.  Included  in  DAR,  IAW  AFLC  Supplement  l,  attachment  6,  to  AFR  300- 12. 

L.  TELECOMMUNICATIONS  REQUIREMENTS  STATEMENT 

1.  Prepared  IAW  AFR  300-12,  Chapter  2,  paragraph  2-9. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Requirement  coordinated  through  HQ  AFLC/DC  with  Air  Force  Com¬ 
munications  Command  (AFCC). 

4.  Included  in  DAR,  IAW  AFLC  Supplement  I,  attachment  6,  to  AFR  300-  12. 

M.  NEW  START/STOP  REQUEST 

1.  Prepared  IAW  AFM  26-1,  Chapter  I  and  AFLC  Supplement  1. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Sent  to  AF/MPMX  (with  copy  to  DAR)  by  AFLC/DP. 


N.  ENVIRONMENTAL  ASSESSMENT  {FORMAL/INFORM AL)/STATEMENT 

1.  Prepared  IAW  AFR  19-2  and  AFLC  Supplement  1. 

2.  Prepared  by  organization  having  the  requirement. 

3.  Certified  by  proponent  of  requirement. 

9.  Accompanies  OAR  through  decision  making  process  to  HQ  USAF. 

O.  STATEMENT  OF  WORK  (SOW) 

1.  Prepared  IAW  AFR  300-12,  Chapter  6,  paragraph  6-2,  and  attachment  23. 

2.  Prepared  by  organization  requiring  contractor  support. 

3.  Sent  with  OAR  and  approved  by  HQ  AFLC/ACD  or  OAR  approval 
authority. 

4.  Sent  to  servicing  contracting  office  or  GSA  (with  GSA  Form  2068)  by  HQ 
AFLC/ACD. 


II.  DOCUMENTS/ ACTIONS  NECESSARY  TO  COMPLETE  THE  CONCEPTUAL  PHASE 

A.  DATA  PROJECT  DIRECTIVE  (DPD) 

1.  DPD  received  from  DAR  approval  authority  (supplemented  by  ADP  Single 
Manager  as  appropriate). 

2.  Prepared  IAW  APR  300-12,  Chapter  2,  paragraph  2-6,  attachment  2,  and 
AFLC  Supplement  1. 

3.  Directed  actions  to  be  taken  by  project  participants. 

B.  DATA  PROJECT  PLAN  (DPP) 

1.  Prepared  IAW  APR  300-12,  attachment  3  and  AFLC  Supplement  1. 

2.  OPR  -  f^oject  Management  Office  (PMO). 

3.  OCR(s)  -  Organizations  participating  in  supporting  project. 

4.  Approved  by  DAR  approval  authority. 

3.  Describes  actions  to  achieve  project  performance,  schedule,  and  cost; 
objectives  specified  in  DPD. 

C.  FUNCTIONAL  DESCRIPTION  (FD) 

1.  Prepared  (when  needed)  IAW  AFR  300-12,  attachment  26. 

2.  OPR  -  Project  Management  Office. 

3.  OCR(s)  -  Organizations  participating  in/supporting  project. 

4.  Baselined  by  project  manager  subsequent  to  System  Requirements  Review 
(SRR). 

5.  Defines  system  requirements  (after  system  analysis)  to  be  satisfied. 

6.  DODS  7935.1 -S  should  be  read  prior  to  starting  preparation  of  the  FD. 

D.  SYSTEM  REQUIREMENTS  REVIEW  (SRR) 

1.  Conducted  IAW  AFR  300-15,  Chapter  3  and  AFLC  Supplement  1. 

2.  Chaired  by  project  manager  of  his/her  designee. 

3.  Attended  by  representatives  of  participating/support  (proiect)  organ¬ 
izations. 

4.  Conducted  to  review/finaiize  contents  of  FD  and  make  sure  all  project 
participants/supporters  have  mutual  understanding  of  the  ADS  require¬ 
ments.  In  addition  the  DAR,  DPD,  and  DPP  must  also  be  reviewed. 
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E.  CONFIGURATION  CONTROL  BOARD  (CCB)  MEETING 


1.  Conducted  IAW  AFR  300-15,  Chapter  2  and  AFLC  Supplement  1. 

2.  Chaired  by  project  manager  or  his/her  designee. 

3.  Attended  by  members  specified  in  DPD  or  DPP. 

4.  Conducted  to  approved  Configuration  Control  Directive  (CCD)  (AFLC 
Form  406)  requesting  initial  baselining  or  to  review /classify  requested 
changes  to  established  baselines. 

F.  PROGRAM  MANAGER  REVIEW  (PMR) 

1.  Conducted  IAW  AFLCR  400-  IS. 

2.  Chaired  by  AFLC/SC  (LM5  program  manager). 

3.  Project  presentations  given  by  project  manager  (or  his/her  designee). 

4.  Attended  by  project  manager,  deputy  project  manager  and  representatives 
from  participating/supporting  organizations. 

5.  Conducted  after  completion  of  a  major  milestone  (or  a  least  every  6 
months)  to  assess  project  progress  and  give  direction 
(continue/change/terminate)  per  AFLC  Form  1298. 

G.  ADPE/PERFORMANCE  SPECIFICATION 

1.  Prepared  IAW  AFR  300-12,  Chapter  9. 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project. 

4.  Approved  by  approval  authority  shown  in  AFR  300-2,  attachment  2 
(contact  HQ  AFLC/ACD  for  verification). 

5.  Delegation  of  Procurement  Authority  requested  frcm  GSA  (if  needed)  by 
approval  authority. 

6.  If  approval  authority  is  HQ  USAF/AC),  RFP  prepared  by  AFC  AC 
(reference  AFR  300-12,  Chapter  9  and  attachment  18). 

7.  If  approval  authority  is  HQ  USAF/ACD,  RFP  prepared  by  servicing 
contracting  office  (reference  AFR  300-  12,  Chapter  9  and  attachment  15). 

H.  STATEMENT  OF  WORK  'SOW)  -  See  entry  I.O. 

I.  DATA  AUTOMATION  REQUIREMENT/ AMENDMENT 

1.  Prepared,  when  needed,  based  on  criteria  in  AFR  300-12,  Chapter  2, 
paragraph  2 -5b. 

2.  See  entry  I.E.  for  remaining  actions/responsibiiities. 


m.  ACTIONS  NECESSARY  TO  BEGIN  THE  DEFINITION  PHASE 


A.  DATA  PROJECT  DIRECTIVE/UPDATE/AMENDMENT 

1.  Prepared,  when  needed,  based  on  criteria  in  AFR  300-1 
paragraph  2-6. 

2.  See  entry  II.A  for  remaining  actions/responsibilities. 

B.  DATA  AUTOMATION  REQUIREMENT/ AMENDMENT 

1.  Prepare  baseline  change  request  forms  as  needed. 

2.  See  entry  H.E  and  11.1  for  remaining  actions/responsibilites. 

C.  STATEMENT  OF  WORK  -  SEE  ENTRY  I.O. 
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IV.  ACTIONS  NECESSARY  TO  COMPLETE  THE  DEFINITION  PHASE 

A.  DATA  PROJECT  PLAN  UPDATE 

1.  PMO  updates  those  parts  of  the  DPP  that  need  updating  as  a  result  of 
changes  in  project  scope  (DAR/DPD  amendments,  etc.),  schedule, 
resources,  benefits,  etc. 

2.  See  entry  n.B  and  AFLCR  400- 13  for  remaining  actions/responsibilities. 

B.  FUNCTIONAL  DESCRIPTION  CHANGES 

1.  PMO  changes  the  FD  as  a  result  of  Baseline  Change  Requests,  which  are 
prompted  by  DAR(S)/DAR  amendments,  SPR(s),  etc.  Update  Configuration 
Management  Status  Accounting  Logs  as  appropriate. 

2.  See  entry  n.C  and  Q.E  for  remaining  actions/responsibilities. 

C.  SYSTEM/SUBSYSTEM  SPECIFICATION  (SS) 

1.  Prepared  (when  needed)  IAW  DOO  Standard  79  35. 1 -S. 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/support  project  (ADS  design  ac¬ 
complished  by  ADP  personnel). 

4.  Baselined  after  the  System  Design  Review  (SDR). 

3.  Defines  the  ADS  design  (system/subsystem/programs)  further  defines  the 
ADS  functions,  and  identifies  (allocates)  the  functions  from  the  FD  to  each 
program. 

D.  OATA  REQUIREMENTS  DOCUMENT  (RD) 

1.  Prepared  IAW  DOD  Standard  7935.1  -S. 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project. 

4.  Baselined  after  the  System  Design  Review. 

5.  Defines  and  lists  data  elements  and  defines  users  data  collection  require¬ 
ments. 

E.  SYSTEM  DESIGN  REVIEW  (SDR) 

1.  Conducted  IAW  AFR  300-13,  Chapter  3  and  AFLC  Supplement  I. 

2.  Chaired  by  project  manager  of  his/her  designee. 

3.  Attended  by  representatives  of  participating/supporting  organizations. 
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a.  Conducted  to  review/finalize  contents  of  SS(s)  and  RD  (when  applicable) 
and  make  sure  proposed  ADS  design  will  satisfy  user's  needs  as  specified 
within  the  FD. 


F.  CONFIGURATION  CONTROL  BOARD  -  See  entry  H.E. 

G.  PROGRAM  MANAGER  REVIEW  -  See  entry  II.F. 


V.  ACTIONS  NECESSARY  TO  BEGIN  THE  DEVELOPMENT  PHASE 

A.  DATA  PROJECT  DIRECT! VE/UPDATE/ AMENDMENT  -  Sec  entry  ll.A. 

B.  DATA  AUTOMATION  REQUIREMENT/ AMENDMENT  -  See  entry  III.B. 

C.  STATEMENT  OF  WORK  -  See  entry  1.0. 


VI.  ACTIONS  NECESSARY  TO  COMPLETE  THE  DEVELOPMENT  PHASE 

A.  DATA  PROJECT  PLAN  UPDATE  -  See  entry  (V.A. 

B.  FUNCTIONAL  DESCRIPTION  CHANGE  -  See  entry  IV.B. 

C.  SYSTEM/SUBSYSTEM  SPECIFICATION/CHANGE 

1.  PMO  changes  the  SS  as  a  result  of  Baseline  Change  Requests,  which  are 
prompted  by  DAR(S)/DAR  amendments,  FD  baseline  changes,  etc. 

2.  See  entry  IV.C.  and  II. E.  for  remaining  actions/responsibilities. 

D.  DATA  REQUIREMENTS  DOCUMENT  CHANGE 

1.  PMO  changes  the  RD  as  required. 

2.  See  entry  IV.D.  and  II.E.  for  remaining  actions/responsibilities. 

E.  DATA  BASE  SPECIFICATION  (DS) 

1.  Prepared  IAW  DOD  Standard  7935.1  -S. 

2.  OPR  *  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project  (Data  base  design 
done  by  ADP  personnel). 

4.  Baselined  after  the  Product  Verification  Review. 

5.  Describes  the  Storage  alloction  and  data  base  (tape,  disk,  etc.) 
organization. 

F.  PROGRAM  SPECIFICATIONS)  (PS) 

1.  Prepared  (when  needed)  IAW  DOD  Standard  7935. 1-S. 

2.  OPR  -  Program  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project  (program  design 
done  by  ADP  personnel). 

4.  Baselined  after  the  Produce  Verification  Review. 

5.  Defines  the  design  of  each  program  in  the  ADS/SS  and  identifies  the 
function(s)  performed  by  the  program(s). 

G.  TEST  PLAN  (PT) 

1.  Prepared  (when  needed)  IAW  DOD  Standard  79 3 5. 1 -S  and  AFLC  Supplement 

I. 

2.  OPR  -  Project  Management  Office. 


3.  OCR  -  Organizations  participating  in/supporting  project  (reference  AFR 
300-13,  Chapter  5  and  AFLC  Supplement  1  for  more  specifics). 

9.  Baselined  after  the  Product  Verification  Review. 

3.  Defines  the  test  program  (plan)  to  be  conducted  by  an  independent  test 
group. 

H.  MAINTENANCE  MANUAL  (MM) 

1.  Prepared  IAV  DOD  Standard  7933. 1-S,  AFLC  Supplement  1  and  AFR 
300-13,  AFLC  Supplement  l. 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project. 

9.  Baselined  after  the  Product  Verification  Review. 

5.  Describes  computer  programs  in  a  detailed  technical  presentation  to  assist 
maintenance  programmer(s). 

I.  USER'S  MANUAL  (UM) 

I 

1.  Prepared  IAV  DOD  Standard  7933.1-S  and  AFLC  Supplement  1. 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project  (prepared  by  the 
ADS  ’’user"  organization(s). 

9.  Published/managed  IAV  AFR  3-9  and  AFR  3-1. 

3.  Provides  the  "user"  organization(s)  management  and  staff  with  information 
and  instructions  for  managing  and  using  the  ADS. 

3.  COMPUTER  OPERATION  MANUAL  <OM) 

1.  Prepared  IAV  DOD  Standard  7933.1-S  and  AFLC  Supplement  l. 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project. 

9.  Baselined  after  to  the  Product  Verification  Review. 

3.  Gives  the  information  needed  by  computer  operations  personnel  to  operate 
the  ADS. 

K.  PRELIMINARY  DESIGN  REVIEW  (PDR) 

1.  Conducted  IAV  AFR  300-13,  Chapter  3  and  AFLC  Supplement  I. 

2.  Chaired  by  project  manager  or  his/her  designee. 


3.  Attended  by  representatives  of  participating/supporting  organizations. 

4.  Conducted  to  review  completed  SS(s),  which  specifies  the  design  of  each 
Computer  Program  Configuration  Item  (CPCI).  Also  to  review  a  draft  of 
the  test  plans  contained  in  the  Test  Plan  (TP). 

L.  CRITICAL  DESIGN  REVIEW  (CDR) 

1.  Conducted  IAW  APR  300-15,  Chapter  3  and  AFLC  Supplement  1. 

2.  Chaired  by  project  manager  or  his/her  designee. 

3.  Attended  by  representatives  of  participating/supporting  (project) 
organizations. 

4.  Conducted  to  review  the  design  of  each  program  within  the  CPCI,  usually 
prior  to  coding.  Other  supporting  documents  are  also  reviewed  (reference 
AFR  300-15  and  AFLC  Supplement  l)  to  make  sure  the  proposed  design 
solution  will  meet  all  requirements. 

M.  PRELIMINARY  FUNCTIONAL/PHYSICAL  CONFIGURATION  AUDIT 

1.  Conducted  IAW  AFR  300-15,  Chapter  3  and  AFLC  Supplement  1. 

2.  Conducted  prior  to  the  Product  Verification  Review. 

3.  Conducted  by  individual(s)/organization(s)  listed  in  the  DPD  (reference 
AFLC  Supplement  l,  paragraph  3-11  to  AFR  300-15.) 

4.  Items  to  be  reviewed  made  available  to  auditors  by  project  manager. 

5.  Results/recommendations  given  to  project  manager  by  auditors. 

6.  Conducted  to  make  sure  the  CPCI  meets  functional  requirements  and  the 
physical  components  are  correct,  complete,  and  consistent  (listings  vs  logic 
charts  vs  narrative  definitions,  etc.). 

N.  PRODUCT  VERIFICATION  REVIEW  (PVR) 

1.  Conducted  IAW  AFR  300-15,  AFLC  Supplement  I. 

2.  Chaired  by  project  manager  or  his/her  designee. 

3.  Attended  by  representatives  of  participating/supporting  organizations. 

4.  Conducted  to  review  all  products  comprising  the  CPCI  and  to  make  sure  all 
necessary  preparations  have  been  made  to  begin  formal  environmental 
testing. 

O.  CONFIGURATION  CONTROL  BOARD  (CCB)  MEETING  -  See  entry  II. E. 

P.  PROGRAM  MANAGER  REVIEW  -  See  entry  II.F. 


vn.  ACTIONS  NECESSARY  TO  BEGIN  THE  TEST  PHASE 


A.  DATA  PROJECT  DIRECTIVE/UPDATE/ AMENDMENT  -  See  entry  III.A. 

B.  DATA  AUTOMATION  REQUIREMENT/AMENDMENT  -  See  entry  III.B. 

C.  STATEMENT  OP  WORK  -  See  entry  I.O. 


VIII.  ACTIONS  NECESSARY  TO  COMPLETE  THE  TEST  PHASE 


A.  DATA  PROJECT  PLAN  UPDATE  -  See  entry  IV.A. 

B.  FUNCTIONAL  DESCRIPTION  CHANGE  -  See  entry  IV.B. 

C.  SYSTEM/SUBSYSTEM  SPECIFICATION  CHANGE  -  See  entry  IV.C. 

D.  DATA  REQUIREMENT  DOCUMENT  CHANGE  -  See  entry  VI.D. 

E.  DATA  BASE  SPECIFICATION  (DS)  CHANGE 

1.  PMO  changes  the  DS  as  a  result  of  Baseline  Change  Requests,  which  are 
prompted  by  DAR(S)/DAR  amendments,  hardware  configuration  changes, 
etc.  2.  See  entry  Vl.E.  and  II. E.  for  remaining  actions/responsibilites. 

F.  PROGRAM  SPECIFICATION  (PS)  CHANGE 

1.  PMO  changes  the  PS(s)  as  a  result  of  Baseline  Change  Requests,  which  are 
prompted  by  DAR(S)/DAR  amendments,  changes  in  functions  performed, 
changes  in  logic,  etc. 

2.  See  entry  VI.F.  and  II. E.  for  remaining  actions/responsibilites. 

G.  TEST  PLAN  (PT)  CHANGE 

1.  PMO  changes  the  PT  as  a  result  of  Baseline  Change  Requests,  which  are 
prompted  by  DAR(S)/DAR  amendments,  changes  in  test  plans,  conditions, 
etc. 

2.  See  entry  VI.G.  and  II.E.  for  remaining  actions/ responsibilities. 

H.  MAINTENANCE  MANUAL  (MM)  CHANGE 

1.  PMO  changes  the  MM  as  a  result  of  Baseline  Change  Requests,  which  are 
prompted  by  DAR(5)/DAR  amendments,  program  changes, 
hardware/software  (basic)  changes,  etc. 

2.  See  entry  VI.H.  and  II.E.,  for  remaining  actions/responsibilites. 

I.  USER’S  MANUAL  (UM)  CHANGES 

1.  PMO  changes  the  UM  IAW  AFR  5-1  (the  contents  of  the  UM  must  remain 
consistent  with  the  other  CPCl  documentation). 

2.  See  entry  VI.!.  and  II.E.  for  remaining  actions/responsibilites. 

3.  COMPUTER  OPERATION  MANUAL  (OM)  CHANGE 

1.  PMO  changes  the  OM  as  a  result  of  Baseline  Change  Requests,  which  are 
prompted  by  OAR(5)/DAR  amendments,  procedural  changes,  program 
changes,  etc. 

2.  See  entry  VI. J.  anc  II.E.  for  remaining  actions/responsibilites. 


K.  TEST  ANALYSIS  REPORT  (RT) 


1.  Prepared  IAW  DOD  Standard  7935.1-S,  AFLC  Supplement  l  and  AFR 
300-15,  AFLC  Supplement  l. 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in  formal  (environmental)  testing. 

9.  Given  to  audit  team  prior  to  final  functionai/physicai  configuration  audit. 

5.  Prepared  to  document  the  results  of  testing  the  AOS  functions  which  are 
needed  to  satisfy  the  '’user's"  requirements.  Also  to  document  known 
deficiencies  and  areas  of  improvement. 

L.  PINAL  PUNCTIONAL/PHYSICAL  CONFIGURATION  AUDIT  (FCA/PCA)  -  See 

entry  VLM.  far  necessary  actions/ responsibilities. 

M.  SYSTEM  VALIDATION  REVIEW  (SVR) 

1.  Conducted  IAW  APR  300-15,  Chapter  3  and  AFLC  Supplement  1. 

2.  Chaired  by  project  manager  or  his/her  designee. 

3.  Attended  by  representatives  of  participating/supporting  organizations  and 
"user"  representative  authorized  and  responsible  for  accepting/rejecting 
developed  ADS. 

4.  Conducted  after  completion  of  formal  environmental  testing  for  purpose  of 
users  certifying  that  ADS  satifies  requirement(s)  and  is  ready  for  use. 

N.  CONFIGURATION  CONTROL  BOARD  (CCB)  -  See  entry  II.E. 

O.  PROGRAM  MANAGER  REVIEW  (PMR)  -  See  entry  II.F. 


IX.  OPERATIONAL  PHASE  ACTIONS 


A.  DATA  PROJECT  DIRECTIVE/UPDATE/ AMENDMENT  -  See  entry  III.A. 

B.  LETTER  OF  TRANSMITTAL  (TL) 

1.  Prepared  IAW  AFR  300-15,  Chapter  3  and  AFLC  Supplement  1  (draft 
reviewed  at  preliminary  FCA/PCA). 

2.  OPR  -  Project  Management  Office. 

3.  OCR  -  Organizations  participating  in/supporting  project. 

4.  Prepared  to  send  material  and  instructions  needed  for  ADS 
testing/implementation. 

C.  FINAL  OPERATIONAL  (EVALUATION)  REVIEW  (FOR) 

1.  Conducted  IAW  AFR  300-12,  Chapter  4. 

2.  Chaired  by  project  manager  or  his/her  designee. 

3.  Attended  by  representatives  of  participating/supporting  organizations  and 
user  representative^). 

4.  Conducted  after  ADS  implementation  to  make  sure  ADS  works 
satisfactorily  after  operating  for  a  period  of  time  in  an  operational 
environment. 

D.  PROGRAM  MANAGER  REVIEW  (PMR)  -  See  entry  II.F. 
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1.  CONFIGURATION  MANAGEMENT  (CM) 


A.  INTRODUCTION 

CM  is  the  implementation  and  maintenance  methodology  of  identification, 
control,  and  status  accounting,  ensuring  effective  control  of  the 
definition/configuration  of  a  desired  product  throughout  the  life  of  the  system. 
The  purpose  of  configuration  management  is  to  ensure  maintainability  of  the 
system  throughout  its  life  cycle. 

The  individual^)  accompli: hing  the  CM  functions  is  responsible  for: 

Identification  and  documentation  of  the  functional  and  physical  charac¬ 
teristics  of  an  item. 

Controlling  changes  to  those  characteristics. 

Recording  and  reporting  status  of  proposed  changes. 

CM  basically  deals  with  the  documentation  specifying  or  reporting  the  status  of 
a  configuration  item  (Cl). 

The  Configuration  Management  Plan  (CMP)  is  an  appendix  of  the  DPP  and  will 
be  prepared  by  the  project  manager.  (AFR  300-15,  AFLC  Sup  1,  para  2.2). 

The  CMP  describes  responsibilities  and  procedures  for  implementing  CM  within 
the  project  and  should  be  completed  prior  to  the  SRR.  (See  AFR  300-1 5,  attch. 
4  for  format). 

The  scope  of  these  procedures  and  number  of  personnel  assigned  CM  respon¬ 
sibilities  should  be  tailored  to  the  quantity,  size,  scope,  stage  of  life  cycle, 
nature  and  complexity  of  the  Cl  involved,  and  whether  it  is  government  or 
contractor  developed  at  government  expense  or  privately  developed  and  offered 
for  government  use. 

A  baseline  is  a  specification,  under  change  control,  defining  a  configuration 
item  (Cl)  at  a  point  in  time.  These  baselines  must  be  maintained  for  the  life  of 
the  system. 

The  functional  baseline  is  defined  by  the  functional  description.  This  is  the 
agreed  to  definition  of  the  system  requirements:  performance,  operational, 
logistical,  training,  etc. 

The  system/subsystem  specifications  define  the  allocated  baseline.  These  are 
performance-level  specifications  which  define  what  functions  are  to  be  per¬ 
formed  by  what  program(s)  or  piece  of  equipment. 

The  program  specifications  define  the  product  baseline.  These  specifications 
define  the  components  (programs)  of  the  operational  system 

A  computer  program  is  a  sequence  of  coded  instructions  performing  a  function 
or  set  of  functions.  It  may  be  a  CPCI  (below)  or  part  of  a  CPCI.  If  a  program 
is  developed  to  assist  the  development  effort,  but  will  not  become  a  component 
of  the  operational  system,  it  is  not  considered  to  be  a  CPC|  or  a  part  of  a  CPC  . 


1  i 


A  computer  program  configuration  item  (CPCI)  is  the  actual  ceded  instruc¬ 
tions  recorded  on  a  storage  medium.  It  may  in  fact  be  a  module,  program,  or  a 
system  comprised  of  programs. 

Computer  program  component  (CPC)  is  a  component  part  of  a  CPCI.  It  may  be 
a  module  or  a  program. 

B.  CONFIGURATION  IDENTIFICATION 

Configuration  Identification  is  the  documentation  describing  the  physical  and 
functional  characteristics  of  an  item.  In  the  case  of  DOD  computer  based 
systems  this  documentation  is  done  IAW  DODS  7935.1 -S.  The  definition  of 
these  characteristics  are  evolved  during  the  acquisition  process  and  the 
identification  is  established  upon  approval  by  the  customer  during  the  on-going 
review  process. 

Designation  of  a  CPCI  is  done  on  the  basis  of  the  engineering  design  taking  into 
consideration: 

Project  Management  Philosophy 
Technical  Risk 
Complexity 
Separate  Computers 
Separate  Schedules 
Different  Functions 

A  CPCI  is  a  level  of  management,  it  is  the  level: 

At  which  the  project  manager  accepts  the  product  -  i.eM  program,  module, 
or  ADS. 

Below  which  the  development  manager  accepts  ail  responsibility. 

Above  which  the  project  manager  is  responsible  for  interface,  integration, 
and  performance. 

Each  CPCI  requires: 

Index 

System/Subsystem  Specification 

BCR(s)  and  Associated  Status  Reports  and  Files 

Separate  Reviews  and  Audits 

User  and  Operating  Manuals 

Formal  Development  and  Operational  Testing 

Formal  Acceptance 
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A  Software  Documentation  Plan  (reference  AFR  300-15,  Chapter  2,  para.  2-7 
and  AFLC  Supplement  l.)  establishes  the  documentation  to  be  produced  and  its 
management.  This  includes  technical  and  management  documentation,  reports 
and  official  letters,  and  standard  forms.  The  following  should  be  covered: 

Title  and  identification  Number 

Purpose 

Who  is  Responsible 

Cordination  and  Approval  Authorities 

Schedule 

Publication  and  Distribution 

Documentation  standards  are  covered  in  OODS  7935.1 -S  and  the  AFLC  supple¬ 
ment. 

C.  CONFIGURATION  CONTROL 

Configuration  Control  is  the  practice  of  maintaining  strict  change  control  over 
requirements,  design,  code,  and  test  activities.  It  is  the  means  of  ensuring 
coordination  between  Project  Management,  Development  Activity  Personnel 
(DAP),  and  Functional  Area  Personnel  (FAP). 

The  Configuration  Control  Board  (CCB)  is  the  management  activity,  chaired  by 
the  project  manager,  which  makes  all  significant  decisions  relating  to  baselined 
documents  and  baseline  change  requests. 

The  Baseline  Change  Request  (BCR)  is  the  recommendation  of  an  alteration  to 
the  configuration/definition  subsequent  to  the  establishment  of  the  baseline. 
The  proposal  includes  the  statement  of  the  requirement  for  the  proposed 
alteration  as  well  as  the  impact  on  cost,  schedule,  and  performance. 

Change  Classifications  include  three  types: 

A  Class  I  change  modifies  an  established  baseline  and/or  impacts  approved 
cost,  schedule  or  performance.  Class  I  changes  must  be  approved  by  the 
CCB  prior  to  implementation. 

A  Class  II  change  is  a  documentation  change  (correction).  It  does  not 
impact  approved  cost,  schedule,  or  performance  and  does  not  require  prior 
approval  by  the  CCB,  only  the  Project  Manager. 

Internal  changes  are  those  made  to  a  CPC1  or  document(s)  prior  to 
baselining. 


NOTE:  In  the  case  of  a  non- government  development  activity,  all  Class  1 
and  11  changes  must  be  concurred  with  by  the  Government  Procurring 
Contracting  Officer  (PCO)  prior  to  implementation. 


D.  CONFIGURATION  STATUS  ACCOUNTING 

Configuration  Status  Accounting  is  the  management  process  of  tracking  a 
change  from  its  official  recording  (BCR,  DPR,  SPR,  DBCR,  etc.)  ixitil  it  is 
disapproved,  or  approved  and  officially  incorporated  into  the  system  and 
associated  documentation.  It  is  also  the  process  of  tracking/monitoring  the 
configuration  of  operational  systems. 

Records  are  maintained  giving  the  current  position  (status)  of  the  item.  These 
record /logs  include: 

Product  Log 

Module  Log 

Software  Problem  Report  Log 
Data  Base  Change  Request  Log 
Baseline  Change  Request  Log 
Design  Problem  Report  Log 

CPCI  Index  describes  the  current  status  of  individual  documents.  It  reports  the 
basic  issue  or  complete  revision  of  each  maintainable  document  and  maintains 
the  current  status  of  each  with  respect  to  approve  BCR(s).  It  should  contain  a 
summary  record  of  dates  for  development  milestones.  The  BCR  log  should  be 
used  in  conjunction  with  the  index  and  must  be  consistent.  The  index  is 
established  within  30  days  of  the  allocated  baseline. 

BCR  Log  is  initiated  following  the  initiation  of  the  first  BCR.  Its  purpose  is  to 
portray  the  status  of  all  BCRs. 

E.  CONFIGURATION  MANAGEMENT  FORMS 

The  Configuration  Control  Directive  (CCD),  AFLC  Form  >06,  will  document  ail 
decisions  of  the  CCB. 

The  Baseline  Change  Request,  AF  Form  1773,  formally  requests  a  baseline 
change.  This  form,  along  with  the  appropirate  attachments,  comprises  the 
proposal  which  is  considered  by  the  CCB. 

The  Design  Problem  Report  (DPR),  AF  Form  1774,  is  used  to  document 
problems  identified  during  reviews  and  audits. 

The  Software  Problem  Report  (SPR),  AF  Form  1775,  is  used  to  document  a 
suspected  or  existing  discrepancev  or  deficiency  in  a  computer  program,  it', 
documentation,  or  interfacing  hardware. 

The  Data  Base  Change  Request,  AF  Form  1776.  is  used  to  request  a  modifi¬ 
cation  to  a  baselined  data  base  and  must  accompany  an  AF  Form  1773  if  an 
established  baseline  is  affected.  DBCR(s)  can  be  submined  at  any  point  in  the 
life  cycle  after  the  document(s)  defining  contents  and/or  struc*ure  of  the  data 
base(s)  have  been  baselined.  Some  changes  may  in  fact  exceed  the  srope  of  the 
DPD  and  this  would  require  the  submission  and  approval  of  a  DAR/PAR 
amendment. 


The  decision  of  the  Project  Manager  can  be  tos 
Reject  the  Request 
Approve 

Approve  With  Specific  Changes  (which  may  be  delayed  implementation 
date) 

BCR(s)  should  includet 
Identification  of  Desired  Change 
Justification 
Summary  of  Alternatives 

Identification  of  Required  Tasks,  Responsible  Agency,  and  Schedule 

Identification  of  Impacts  on  Schedule,  Cost,  and  Other  System  s/C  PC  I(s) 

The  AFLC  Form  406  is  used  for  coordination  and  to  reflect 
approval/disapproval  of  a  Class  I  change. 

The  approved  edification  to  the  baselined  document  is  done  via  a  change  or 
modification. 

A  change  is  a  package  of  pages  which  have  been  modified. 

A  revision  is  a  complete  reissue.  A  revision  is  required  when  over  40%  of  a 
document  has  changed  or  will  be  changed. 

The  revision  is  prepared,  issued,  and  identified  the  same  way  the  original 
document  was.  The  revision's  specification  number  will  contain  a  revision 
letter. 

Change  pages  will  contain  specification  numbers  and  date. 

Specification  numbers  are  assigned  by  AFLC/ACTM. 

Specification  numbers  must  be  on  each  page  in  the  upper  corner  opposite  the 
binding  edge. 

Version  numbers  appear  only  on  the  CPCI  and  associated  date/documentation 
from  agency  to  agency. 

CM  APPLICATION  OF  CONFIGURATION  MANAGEMENT  TO  ADS 
DEVELOPMENT 

Purposes  to  outline  method  by  which  HO  AFLC/ ACT  will  implement 
management  audits  of  selected  AFLC  LMS  development  projects. 

Configuration  management  k  procedures  for  application  (AFR  WO- 1  *>, 
chap.  2). 


Reviews  and  audits  in  Configuration  Management  (AFR  300-15,  chap.  3). 

Specific  policy  and  procedures  for  application  of  CM  within  AFLC  (AFR 
300-15,  AFLC  Supp  1). 

Authority  for  AFLC/ ACT  to  conduct  audits  (AFR  300-15,  AFLC  Supp  1,  para 
2-22b*  AFLCR  *00-18,  para  2-10g)  including  review  of  DPDs  and  DPPs. 

Reference!  AFR  300-15,  AFLC/AC  O!  171-6. 
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111.  QUALITY  ASSURANCE 

Quality  Assurance  (QA)  is  the  implementation  of  a  series  of  planned  activities 
whose  objective  is  to  provide  confidence  that  the  end  products  conform  to 
established  technical  requirements  and  performance  standards. 

A  QA  Plan  is  produced  based  on  a  review  of  the  DAR,  DPD,  DPP,  and  FD.  The 
plan  describes  QA  functions,  responsibilities,  and  tools  (reference  AFR  900-15, 
Chapter  t  and  AFLC  Supplement  1). 

The  QA  Program  impacts  Configuration  Management,  Testing,  Data  Manage¬ 
ment,  Technical  Standards,  Program  Design  and  Coding  Standards,  Reviews  and 
Audits. 

Software  QA  must  be  life-cycle-oriented  to  reduce  maintenance  costs. 

There  is  not  an  AFLC  software  quality  assurance  program.  Therefore  it  is  the 
responsibility  of  the  project  manager  to  ensure  a  "Quality  Product"  is  delivered. 


IV.  MCAP 


HQ  USAF/ACD  requires  a  new  MCAP  each  year  by  1  Oct.,  six  copies  (APR 
300-7,  para  Sd(l). 

The  plan  covers  the  four  year  period  beginning  two  years  after  date  submitted 
(APR  300-7,  para  S<Xl). 

Its  purpose  is  to  document  ADP  requirements  as  an  aid  to  planning,  budgeting 
and  management  (APR  300-7,  para  3a, b). 

AFLC  Management  Systems  Panel  will  review  and  approve  the  MCAP  subject  to 
review  by  AFLC  Planning  and  Programming  Review  Board  (AFR  300-7,  APLC 
Sup  1,  para  3e  and  Sc(4Xc)). 

HQ  AFLC/ACDR  is  the  focal  point  for  MCAP  policies  and  procedures.  They 
prepare  the  draft  plan  and  distribute  the  approved  plan. 

HQ  AFLC/ACD  will  prepare  MCAP  Sect.  1  Command  Situation  (AFR  300-7, 
AFLC  Sup.  1,  para  ScftHb)  and  forward  to  arrive  in  ACDR  not  later  than  13 
July  (AFR  300-7,  AFLC  Sup  1,  para  8d(5». 

HQ  AFLC/ACDS  will  prepare  MCAP  Sect.  2  ADPS  Plan  Summaries  (AFR  300-7, 
AFLC  Sup  1,  para  8c(4XbX2))  and  forward  to  arrive  in  ACDR  not  later  than  13 
July  (AFR  300-7,  AFLC  Sup  1,  para  3d(5». 

HQ  AFLC/ACD  will  prepare  Sect.  3  and  4,  Projected  Automation  Requirements 
(AFR  300-7,  AFLC  Sup  1,  para  *c(4Xb)3a  thru  3d). 

Preparation  of  PARs  is  continuing  function.  PARs  received  by  13  July  will 
included  in  the  MCAP  for  the  same  calendar  year  (AFR  300-7,  AFLC  Sup  1, 
para  8d(4)). 

Corrections  to  MCAP  required  by  ACD  will  be  effected  within  21  days  from 
receipt  of  notice.  Other  changes  or  updates  will  be  submitted  21  days  before 
effective  date  (AFR  300-7,  para  8d(2)).  These  changes  will  be  approved  by 
AFLC  Management  Systems  Panel;  sufficient  leadtime  must  be  allowed  (AFR 
300-7,  AFLC  Sup  1,  para  *d<2)). 
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V.  EA/M1LESTONE  REPORT 

A.  AF  FORM  2053 

Economic  Analysis  (Summary  of  Alternatives) 

Purpose:  to  compare  the  net  costs  for  each  alternative,  excluding  baseline 
and  augmented  current  system. 

Get  discount  and  uniform  annual  rates  from  AFR  178-1. 

Get  Net  Annual  Costs  from  AF  Form  205*. 

Reference:  AFR  500-12,  para  3- 10a,  3-11. 

Example:  AFR  300-12,  Attachment  7. 

B.  AF  FORM  205* 

Economic  Analysis  (Summary  of  Alternative  No. _ ) 

Purpose:  to  compute  net  cost  of  an  alternative,  even  the  baseline  and  aug¬ 
mented  system. 

Obtain  the  data  for: 


Part 

Form 

I 

AF  Form  2055 

HD. 

AF  Form  2056 

IV 

AF  Form  2057 

Aggregate  parts  I  and  11  to  form  part  III. 

Combine  parts  01  and  IV  to  form  part  V, 

Part  V  becomes  the  input  to  AF  Form  2053. 

References:  AFR  300-12,  para  3- 10b,  3-12. 

Example:  AFR  300-12,  Attachment  8. 

C.  AF  FORM  2055 

Economic  Analysis  (ADPM15  Costs) 

Purpose:  to  compute  the  detailed  ADPMI5  Costs  of  the  baseline  and  each 
alternative. 
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Use  a  separate  form  for  each  phase  of  the  life  cycle,  (AFR  300-12,  para 
3-3). 

Use  a  separate  form  for  each  DP!,  Program  Element  Code  (PEC),  Com¬ 
mand,  or  Appropriation  if  more  than  one  are  reportable. 

Get  coat  categories  from  RCS:  DD-COMP(AR)996. 

Get  salary  factor  guidance  from  ACD  and  include  the  schedule  of  factors 
identified  by  source  and  date  with  each  analysis.  If  using  average  rates  for 
personnel  significantly  affects  choice  between  alternatives,  (APR -300- 12, 
para  3-  13d). 

Line  H,  the  "net"  ADPMIS  COSTS,  becomes  the  input  to  Section  l,  AF  Form 

2054. 

References:  AFR  300-12,  para  3-106,  3-13. 

Examples:  AFR  300-12,  Attachment  9. 

D.  AF  FORM  2056 

Economic  Analysis  (Non-MIS  Costs) 

Non-MIS  costs  include  TDY,  training,  and  others  not  directly  identified 
with  ADP  but  which  are  directly  related  to  new  systems  development. 

Report  costs  in  thousands  of  dollars. 

Use  a  separate  form  for  each  phase  of  the  life  cycle. 

Use  a  separate  form  for  each  DPI,  Program  Element  Code,  Command,  or 
Appropriation  if  more  than  one  are  reportable. 

Cross  reference  costs  elements  with  the  milestone  reporting  system,  AF 
Form  2060. 

'  Get  salary  factor  guidance  from  ACD  and  include  the  schedule  of  factors 
identified  by  source  and  date.  If  using  avarage  rates  for  personnel 
significantly  affects  the  choice  between  alternatives  (AFR  300-12,  para 
3- 13d). 

Give  unique  costs  on  line  D  and  explain  in  a  footnote  or  addendum. 

Line  E,  the  total  Non-MIS  costs,  becomes  the  input  to  Section  IV,  AF  Form 
2054. 

Reference:  AFR  300-12,  para  3- 10c,  3-14. 

Example:  AFR  300-12,  Attachment  10. 

E.  AF  FORM  2057 

Economic  Analysis  (Cost -Reduction  Savings) 

Purpose:  to  report  the  savings  resulting  from  implementing  an  alternative. 
See  AFR  300-12.  para  3-9(2)  for  discussion  of  costs  and  savings. 


Use  a  separate  form  for  each  DPI,  Prog-am  Element  Code  (PEC),  Com¬ 
mand,  or  Appropriation  if  more  than  one  are  reportable. 

Get  data  for  Part  I  from  AF  Form  2055  for  baseline  system. 

Calculate  Part  II,  Augmentation  Avoided.  Get  data  from  AF  Form  2055 
for  augmented  system  and  subtract  cost  of  baseline  system  obtained  above. 

Include  in  Part  III  any  other  savings  that  can  be  identified  to  specific 
budgets  or  major  programs. 

Lines  I,  II,  and  III  B3  become  inputs  to  Section  IV,  AF  Form  2054. 
References:  AFR  300-12,  para  3-I0c,  3-15. 

Example:  AFR  300-12,  Attachment  11. 

F.  AF  FORM  2058 

Milestone  Reporting  System  (Schedule) 

Purpose:  to  describe  the  milestones  for  an  ADPS/ADS  project. 

All  dates  are  given  in  six  digit  form  (YYMMDD). 

Key  milestones  and  their  codes  are  given  in  AFR  300-12,  para  3-2 ld( I). 

Include  other  reviews/audits  specified  in  DPD,  e.g.,  SDR,  PDR,  CDR.  Do 
not  use  codes  except  as  above. 

A  short  title  may  be  given  under  "Remarks" 

Note  the  identification  of  the  rows  on  this  form  with  the  columns  on  AF 
Forms  2059  and  2060. 

References:  AFR  300-12,  para  3-18,  3-21. 

Example:  AFR  300-12,  Attachment  12. 

G.  AF  FORM  2059 

Milestone  Reporting  System  (ADPMI5  Costs) 

Purpose:  to  compute  the  variance  analysis  by  cost  element  for  ADPMIS 
costs. 

Note  the  cross  reference  of  cost  categories  with  AF  Form  20  55. 

Each  report  must  show  costs  for  the  current  milestone,  the  next  three 
milestones  and  the  fully  operational  milestone. 


Columns  O  and  H  record  cumulative  time  and  costs  between  milestones. 
Column  D  gives  actual  cost  to  date;  column  H  projects  total  cost. 

Specific  details  for  each  column  are  given  in  A FR-300- 12,  para  3-22. 

Report  is  due  not  later  than  30  days  after  the  scheduled  milestone  date.  If 
actual  completion  date  will  be  later  than  scheduled  date,  notify  HQ 
USAF/ACDC  of  the  reason  and  the  anticipated  submission  date. 

Management  reviews  are  required  if  the  incremental  cost  exceeds  the 
approved  cost  by  23%  or  the  milestone  slips  120  days.  (A FR -300- 12,  para 
3-23). 


H.  AF  FORM  2060 

Milestone  Reporting  System  (Non-MIS  Costs) 

Purpose:  to  compute  the  variance  analysis  by  cost  element  for  Non- MIS 
costs. 

Note  the  cross  reference  of  cost  categories  With  AF  Form  2056. 

Each  report  must  show  costs  for  the  current  milestone,  the  next  three 
milestones  and  the  fully  operational  milestone. 

Columns  D  and  H  record  cumulative  time  and  costs.  All  other  columns 
reflect  incremental  costs  between  milestones.  Column  O  gives  actual  cost 
to  date;  column  H  projects  total  cost. 

Specific  details  for  each  column  are  given  in  A  FR-300- 12,  para  3-22. 

Report  is  due  not  later  than  30  days  after  the  scheduled  milestone  date.  If 
actual  completion  date  will  be  later  than  scheduled  date  notify 
HQ  USAF/ACDC  of  the  reason  and  the  anticipated  submission  date. 

Management  reviews  are  required  if  the  incremental  cost  exceeds  the 
approved  cost  by  25%  or  the  milestone  slips  120  days.  (AFR-300-12,  para 
3-23). 


VI.  COMMUNICATIONS  CONSIDERATION 


Telecomm  memo  entry  is  made  to  PAR  and  the  Telecommunications  Plan 
(C3P2). 

During  DAR  preparation  Comm  Requirement  is  determined  and  defined  to 
AFLC/DCO  IAW  AFR  300-12,  AFLC  Supp  I,  Atch  6,  para  30.  (OPR -Project 
Manager) 

Communications  alternatives  examined  for  cost  (OPR:  AFLC/DCO) 

A  communications  concept  is  selected  and  discussed  with  Project  Manager 
(AFLC/DC) 

A  Communication  Annex  is  prepared  IAW  AFR  100- IS  for  inclusion  in  the  DAR 
or  DPP  (OPR:  AFLC/DCO) 
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APPENDIX  G 

GLOSSARY  OF  TERMS  FOR  SOFTWARE  CONFIGURATION  MANAGEMENT 

(Ref  141) 


GLOSSARY  OP  TERMS  POR  CONFIGURATION  MANAGEMENT* 


Acceptance 


Acceptance  Testing  - 
Subset  of  Qualification 


Approved  Change 


Audits 


An  official  act  by  the  customer  to  accept 
transfer  of  accountability,  title,  and 
delivery  of  a  Configuration  Item  or  other 
items  on  a  Contract  (e.g.,  Data  Item). 

The  formal  inspection,  testing,  and/or 
Tests  analysis  accomplished  in  accordance  with 
Section  4  of  a  Product  Specification  to 
verify  the  performance  and  adequacy  of 
a  production  item. 

A  change  for  which  approval  has  been 
given  for  change  incorporation  by  the 
applicable  change  control  board. 

Configuration  audits  verify  conformance 
to  specifications  and  other  contract 
requirements.  Audits  are  not  reviews. 
NOTE:  Audita  differ  from  reviews  in 
that  reviews  are  conducted  on  a  periodic 
baeia  to  assess  the  degree  of  completion 
of  technical  efforts  related  to  identified 
milestones  before  preceding  with  further 
technical  effort. 

a.  Functional  Configuration  Audit  (FCA) . 
The  formal  examination  of  functional 
characteristics'  test  data  fot  a 
configuration  item,  prior  to  accept¬ 
ance,  to  verify  that  the  item  uas 
achieved  the  performance  specified 

in  its  functional  or  allocated  con¬ 
figuration  identification. 

b.  Physical  Configmation  Audit  (PC/*. ) . 
The  formal  examination  of  the  "as- 
built"  configuration  of  a  unit  of  a 
Cl  against  its  technical  documenta- 
tion  in  order  to  establish  the  Cl’s 
initial  product  configuration  iden¬ 
tified  ion. 


•This  list  is  by  no  means  complete  but  contains  those  terms  must  frequently 
used  in  the  industry.  The  list  has  bet  r  compiled  l'»on  a  variety  of  industry 
and  government  sources. 


Baseline  A  configuration  identification  document 

or  a  set  of  such  documents  formally 
designated  and  fixed  at  a  specific  time 
during  a  Cl's  life  cycle.  Baselines, 
plus  approved  changes  from  those  base¬ 
lines,  constitute  the  current  configura¬ 
tion  identification.  FOr  configuration 
management,  there  are  three  baselines,  as 
follows : 

a.  Functional  Baseline.  The  initial 
approved  functional  configuration 
identification. 

b.  Allocated  Baseline.  The  initial 
approved  allocated  configuration 
identification. 


Computer  Equipment 


Computer  Firmware 

Coof>uter  Program 


c.  Product  Baseline.  The  initial 

approved  or  conditionally  approved 
product  configuration  identification. 

Devices  capable  of  accepting  and  storing 
conputer  data,  executing  a  systematic 
sequence  of  operations  on  con^uiter  data 
or  producing  control  outputs.  Such 
devices  can  perform  substantial  inter¬ 
pretation,  computation,  comnunication, 
control,  and  other  logical  functions. 

A  microcoded  control  program  that  resides 
in  a  Programnable  Read  Only’  Memory  (PROM) 
or  Read  Only  Memory  (ROD  . 

A  series  of  instructions  or  statements  in 
a  form  acceptable  to  computer  equijwient, 
designed  to  cause  the  execution  of  an 
operation  or  series  of  operations. 

Computer  programs  include  such  items  as 
operating  systems,  assemblers,  compilers, 
interpreters,  data  management  system, 
utility  programs,  and  maintenance/diag¬ 
nostic  programs.  Thev  abo  include 
applicable  programs  such  as  payroll, 
inventory  control,  operational  flight, 
strategic,  tactical,  automatic  test,  crew 
simulator,  and  engineering  analysis 
program^.  Computer  programs  mav  he 
either  machine  dependent  machine 
independent,  and  mav  h<-  general  purpose 
in  nature  or  he  !<■ ;  1  ■'  t,'<'*ati<fy  the 

requirements  ol  a  spei  i  1 1  1  4fd  process  of 
a  part iculai  usei . 


Conputer  Program  Component  (CPC)  A  CPC  is  a  functionally  or  logically 

distinct  part  of  a  computer  program 
configuration  item  (CPCI),  distinguished 
for  purposes  of  convenience  in  designing 
and  specifying  a  complex  CPCI  as  an 
assembly  of  subordinate  elements. 


Computer  Program  Configuration 
Item  (CPCI) 


Computer  Program  Library 


Conputer  Resources 


Configuration 


Configuration  Control 


An  aggregate  of  computer  programs,  or 
of  the  discrete  portions  (e.g.,  CPCs, 
modules,  routines,  etc.)  thereof,  which 
satisfy  an  end  use  function  and  which  is 
designated  by  the  Government  for  con¬ 
figuration  management.  CPCIs  may  vary 
widely  in  conplexity,  size,  and  type, 
from  a  special  purpose  diagnostic  program 
to  a  large  conmand  and  control  system. 

A  controlled  collection  of  program  source 
statements  on  media  such  as  punched  cards, 
magnetic  tapes,  or  discs.  This  library 
is  also  the  repository  for  the  controlled 
master  copies  of  each  computer  program. 

The  totality  of  computer  equipment, 
conputer  program,  computer  data,  associated 
documentation,  personnel,  and  sipplies. 

The  functional  and/or  physical  character 
istics  of  hardware/software  as  set  forth 
in  technical  documentation  and  achieved 
in  a  product. 

The  systematic  evaluation,  coordination, 
approval  or  disapproval,  and  implementa¬ 
tion  of  all  approved  changes  in  the  con¬ 
figuration  of  a  CI/CPCI  after  formal 
establishment  of  its  configuration 
identification. 


Configuration  Control  Board  A  board  conposed  of  representatives  from 

program/project  functional  areas  such  as 
engineering,  configuration  management, 
procurement,  production,  test  and  logistic 
support,  training  activities,  and  using/ 
supporting  organizations.  This  board 
approves  or  disapproves  proposed  engi¬ 
neering  changes  with  each  member  recording 
his  organization's  official  position. 

The  program/project  manager  is  normally 
the  board  chairman  and  makes  the  final 
decision  on  all  changes  unless  otherwise 
directed  by  comnand  policy.  The  board 
issues  a  directive/ request  to  implement 
its  decision. 


ti-‘-  r  m 


Configuration  Identification 


The  current  approved  or  conditionally 
approved  technical  docunentation  for  a 
configuration  item  as  set  forth  in 
specifications,  drawings,  and  associated 
lists,  and  documents  referenced  therein. 

Configuration  Item  (Cl)  An  aggregation  of  hardware/ software,  or 

any  of  its  discrete  portions,  which 
satisfies  an  end  use  function  and  is 
designated  by  the  Government  for  config¬ 
uration  management.  CIs  may  vary  widely 
in  complexity,  size,  and  type,  from  an 
aircraft,  electronic  or  ship  system  to  a 
test  meter  or  round  of  ammunition. 

During  development  and  initial  production, 
CIs  are  only  those  specification  items 
that  are  referenced  directly  in  a  contract 
(or  an  equivalent  in-house  agreement). 
During  the  operation  and  maintenance 
period,  any  reparable  item  designated  for 
separate  procurement  is  a  configuration 
item. 

Configuration  Management  A  discipline  applying  technical  and 

administrative  direction  and  surveillance 
to  (1)  identify  and  dociment  the  functional 
and  physical  characteristics  of  a  config¬ 
uration  item,  (2)  control  changes  to  those 
characteristics,  and  (3)  record  and  report 
change  processing  and  implementation 
status . 

Configuration  Status  Accounting  The  recording  and  reporting  of  the  infor¬ 
mation  that  is  needed  to  manage  config¬ 
uration  effectively,  including  a  listing 
of  the  approved  configuration  identifica¬ 
tion,  the  status  of  proposed  changes  to 
configuration,  and  the  implementation 
status  of  approved  changes. 

Contract  The  legal  agreement  between  DOP  and 

industry,  or  similar  internal  agreement 
wholly  within  the  government ,  for  the 
development,  production,  maintenance,  or 
modification  of  an  item(s). 
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Contract  Data  Requirements 
LIST  (CDRL) 


Contractor 


Cost 

Non-Recurring  Costs 

Recurring  Costs 

Critical  Item 

Critical  Design  Review 


A  contract  form,  DD  1423,  listing  all 
technical  data  items,  selected  from  an 
Authorized  Data  List,  to  be  delivered 
under  the  contract. 

An  individual,  partnership,  company, 
corporation,  or  association  having  a 
contract  with  the  procuring  activity  for 
the  design,  development,  design  and 
manufacture,  manufacture,  maintenance, 
modification  or  supply  of  items  under 
the  terms  of  a  contract.  A  govemnent 
activity  performing  any  or  all  of  the 
above  actions  is  considered  to  be  a 
contractor  for  configuration  management 
purposes. 

The  term  "cost"  means  cost  to  the 
Government . 

One-time  costs  which  will  be  incurred  if 
an  engineering  change  is  ordered  and 
which  are  independent  of  the  quantity  of 
items  changed,  such  as:  cost  of  redesign, 
special  tooling,  or  qualification. 

Cost  which  are  incurred  for  each  item 
changed  or  for  each  service  or  documented 
ordered. 

An  item  within  a  configuration  item  (Cl) 
which,  because  of  special  engineering  or 
logistic  considerations,  requires  an 
approved  specification  to  establish 
technical  or  inventory  control  at  the 
component  level . 

(CDR)  This  review  shall  be  conducted  by  the 

developer  for  each  configuration  item 
when  detail  design  is  essentially  complete. 
The  purpose  of  this  review  will  he  to 
determine  that  the  detail  design  of  the 
configuration  item  under  review  satisfies 
the  design  requirements  established  in 
the  configuration  item  specification, 
and  establishes  the  exact  interface 
relatioaships  between  the  configuration 
item  and  other  items  of  equipment  and 
facilities. 
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fifet*  (Technical 
Information) 


Data  Package 
Deccrani ss ioning 
Debugging 

Deficiencies 

DOD  Components 

Documentation 

DSARC 

Embedded 


■  >r.  ItAV'  -iA 


Data  and  The  means  for  comnunication  of  concepts, 

plans,  descriptions,  requirements,  and 
instructions  relating  to  technical 
projects,  material,  systems,  and  services. 
These  may  include:  specifications, 
standards,  engineering  drawings,  associ¬ 
ated  lists,  manuals,  and  scientific  and 
technical  reports.  They  may  be  in  the 
form  of  docunents,  displays  and  sound 
records,  punched  cards,  and  digital  or 
analog  data. 

A  collection  of  data  products  (items) 
which  is  complete  for  a  specific  use. 

Change  in  program  status  from  operational 
to  inactive. 

A  process  to  detect  and  ranedy  inadequacies, 
preferably  prior  to  formal  testing  and 
operational  use. 

Deficiencies  consist  of  two  types : 
conditions  or  characteristics  in  hardware/ 
software  which  are  not  in  compliance  with 
a  specified  configuration,  or  inadequate 
(or  erroneous)  configuration  identifica¬ 
tion  which  has  resulted  or  may  result,  in 
configuration  items  that  do  not  fulfill 
approved  operational  requirements. 

Term  which  describes  the  Office  of  the 
Secretary  of  Defense,  the  Military 
Departments,  Office  of  the  Joint  Chiefs 
of  Staff,  and  the  Defense  Agencies. 

The  specifications,  reports,  plans  and 
procedures  u*  ed  to  document  and  support 
computer  programs. 

The  Defense  Systems  Acquisition  Review 
Council;  an  advisory  bodv  to  the 
Secretary  of  Defense  on  major  system 
acquisition  programs  and  related  policies. 

Adjective  modifier;  integral  to,  from  the 
design,  procurement,  and  operations  point 
of  v^ew,  espoused  in  DOD  Directive  5000.1. 
For  computer  programs,  this  implies  those 
program  which  are  integral  to,  hut  part 
of  p  larger  system. 


Engineering  Change  An  alteration  in  the  configuration  of  a 

configuration  item  or  an  item,  delivered 
or  to  be  delivered,  or  undeT  development, 
after  formal  establishment  of  its  con¬ 
figuration  identification.  Changes  may 
be  Class  I  or  II . 


Engineering  Change  Priorities 


Engineering  Change  Proposal 


ECP  Types 


Form,  Fit,  and  Function 


The  rank  assigned  to  a  Class  I  engineering 
change,  which  determines  the  methods  and 
resources  to  he  used  in  review,  approval, 
and  implementation.  There  are  three 
recognized  priorities: 

-  Emergency 

-  Urgent 

-  Routine 

A  term  which  includes  both  a  proposed 
engineering  change  and  the  documentation 
by  which  the  change  is  described  and 
suggested. 

A  term  covering  the  subdivision  of  ECPs 
on  the  basis  of  the  completeness  of  the 
available  information  delineating  and 
defining  the  engineering  change,  i.e., 
preliminary  or  formal  ECP. 

That  configuration  comprising  the 
physical  and  functional  characteristics 
of  the  item  as  entity  but  not  including 
any  characteristics  of  the  elements 
making  up  the  item. 


Formal  Qualification  Review  (FQR)  The  FQR  is  a  review  to  ensure  that  the 

quality  assurance  tests  have  been  accomp¬ 
lished  that  verify  that  the  item  as 
designed  performs  as  required  by  the 
specification  performance  requi rements . 

An  FQR  is  held  for  each  new  design 
Computer  Program  Configuration  Item. 
fMU-STD-483) 


Function  A  discrete  action  required  to  achieve  a 

given  objective,  to  be  accomplished  by 
hardware,  computet  program,  personnel, 
facilities,  procedural  data,  or  a 
combination  thereof.  It  is  an  operation 
the  system  must  perform  in  ordri  to  ful¬ 
fill  its  intended  mission. 


Functional  Area 


Awctional  Characteristics 


Functional  Configuration 
Identification  (FCI) 


Hardware/Software 


Integrated  Logistic  Support 


A  distinct  group  of  system  perform-ince 
requirements  which,  together  with  all 
other  such  groupings,  form  the  next 
lower  level  breakdown  of  the  system  on 
the  basis  of  function. 

Quantitative  performance,  operating  and 
logistic  parameters  and  their  respective 
tolerances.  Functional  characteristics 
include  all  performance  parameters,  such 
as  range,  speed,  lethality,  reliability, 
maintainability,  safety. 

The  current  approved  technical  documenta¬ 
tion  for  a  configuration  item  which 
prescribes:  (1)  all  necessary  functional 
characteristics,  (2)  the  tests  required 
to  demonstrate  achievement  of  specified 
functional  characteristics,  (3)  the 
necessary  interface  characteristics  with 
associated  Cls,  (4)  and  CIs  key  functional 
characteristics  and  it?  key  lower  level 
CIs,  if  any,  and  (5)  design  constraints, 
such  as  envelope  dimensions,  component 
standardization,  use  of  inventory  items, 
integrated  logistics  support  policies. 

Hardware  or  software,  or  a  combination  of 
both,  in  which  the  software  includes  only 
that  associated  with  hardware  foi  opera¬ 
tional  use,  e.g.,  computer  programs  for 
command  and  control,  handbooks  for  opera¬ 
tions,  maintenance,  etc. ,  and  excludes 
fabrication  specifications,  drawings,  etc. 

A  composite  of  the  clement?  necessary 
to  assure  the  effective  and  economical 
support  of  a  system  or  equipment  at  all 
levels  of  maintenance  for  its  program 
ming  life  cycle.  The  elements  include 
all  resources  necessary  to  maintain  and 
operate  an  equipment  or  weapon?  system, 
and  arc  categorized  as  follows 
(1)  planned  maintenance,  (2)  logistic 
support  personnel,  (3)  technical  logistic 
data  and  information,  (l)  -uppcit  equip¬ 
ment,  f 5)  span'?  and  repair  pair  , 

(6)  facilities,  and  C)  cent  rat  t  main 
tenance. 


I tan  (When  the  term  is  used 
without  a  modifier) 


Interface  Control  Working 
Group  (ICWG) 


Master 


Mb int enance/Modi f icat ion 
to  a  Program 


Mnemonic 


Object  Form 


Operational  Systems  Development 


Physical  Characteristics 


Any  level  of  hardware  assembly  below  a 
systan;  i.e. ,  subsystem,  equipment, 
component,  subasserbly,  or  part.  Also 
see  "Configuration  Item”,  "Critical 
Item",  and  Privately  Developed  Item". 

A  group  chartered  as  the  official  channel 
among  contractors,  the  customer,  and 
other  agencies  to  resolve  interface 
problems,  exchange  new  interface 
information,  document  change  agreements, 
and  coordinate  Engineering  Change 
Proposals  affecting  interfaces  within 
a  project. 

The  official  version  of  a  docunent, 
system,  or  program. 

The  maintenance  of  a  computer  program  is 
defined  to  be  any  modification  to  the 
instruction  of  an  operating  digital 
computer  program  or  system  for  corrective 
or  improvement  purposes. 

A  brief  alphabetical  title  used  to  refer 
to  a  program,  subprogram,  subroutine, 
etc.  It  often  gives  information  about 
the  program  function  or  serves  as  a 
memory  aid  in  this  regard. 

Coded  instructions  that  can  be  acted 
upon  directly  by  a  computing  system 
without  requiring  an  additional  proc¬ 
essing  step. 

Includes  a  research  and  development 
effort  directed  toward  development, 
engineering,  and  test  of  systems, 
support  programs,  vehicles,  and  weapons 
that  have  been  approved  for  production 
and  Service  Employment. 

Qjantitative  and  qualitative  expressions 
of  material  features,  such  as  compo 
sition,  dimensions,  finishes,  form, 
fit  and  their  respective  tolerances. 


?11 


Preliminary  Design  Review  (PDR)  This  review  shall  be  conducted  for  each 

configuration  item  prior  to  the  detail 
design  process  to:  evaluate  the  progress 
and  technical  adequacy  of  the  selected 
design  approach;  determine  its  compat¬ 
ibility  with  the  performance  require 
ments  of  the  configuration  item 
Development  Specification;  and  establish 
the  existence  and  compatibility  of  the 
physical  and  functional  interfaces 
between  the  configuration  item  and  other 
items  of  equipment  of  facilities. 

(MIL- STD- 1521) 

Product  A  software  product  is  a  coiiipjter  program 

and  its  associated  documentation.  It 
■ay  be  in  the  form  of  card  decks,  mag¬ 
netic  tapes,  disk- packs,  or  other 
physical  media  capable  of  transmitting 
its  contents  directly  to  a  computer. 

Privately  Developed  Item  An  item  completely  developed  at  private 

expense  and  offered  to  the  Government 
as  a  production  article,  with  Government 
control  of  the  article’s  configuration 
normally  limited  to  its  form,  fit,  and 
function. 

A  project  WBS  is  defined  as  the  complete 
WBS  for  the  project,  containing  all  WBS 
elements  related  to  the  development 
and/or  production  of  the  defense  material 
item.  The  project  WBS  evolves  from  the 
project  summary  WBS  extended  to  include 
all  contract  WBS(s)  and  equivalent  WBS ^ s' 
resulting  from  DOD  in-house  efforts. 

The  WBS  is  used  for  cost  accounting  and 
monitoring  of  the  project. 

rhe  transfer  of  physical  custody  and 
control  of  products  or  docum-' itation 
from  the  originator  to  another  organi ra¬ 
tion  in  a  controlled  envi ronm  'nt ,  such 
as  through  a  formal  release  ^  -stem  or 
organi nation. 


',’78 


Project  Work  Breakdown 
Structure  (Project  WBS) 


Release 


A0-AU9  Mi  AIR  FORCE  INST  Of  TECH  WRISHT-PATTERSON  AFB  OH  SCHOO— ETC  f/%  9/2 

SOFTWARE  WUALXTY  NETRICSl  A  SOFTWARE  MANAOEMENT  MONITORINO  METH**CTC(U> 
MAR  At  S  J  JARZOMKK 

UNCLASSIFIED  AFIT/ACS/MA/AEM-1  NL 


Retrofit  (Retroactive  Retrofit) 


Software 


Software  Engineering 


Software  Data  Package 


Source  Form 


Software  Reliability 


Modification  of  a  configuration  item 
to  incorporate  changes  made  in  later 
production  of  a  similar  type. 

A  combination  of  associated  computer 
programs  and  computer  data  required  to 
enable  the  computer  equipment  to  perform 
computational  or  control  fucntions. 

Science  of  design,  development,  imple¬ 
mentation,  test,  evaluation,  arid  main¬ 
tenance  of  computer  software  over  its 
life  cycle. 

A  compilation  or  physical  grouping  of 
all  required  docunentation:  listings, 
card  decks,  tapes,  etc.,  to  support  a 
computer  program. 

Coded  instructions  written  in  one  of 
several  programming  languages  that 
cannot  be  directly  acted  upon  by  a 
computing  system  without  requiring  an 
interim  processing  conversion  to  a 
machine  language  (also  see  "Object  Form") . 

The  probability  that  a  computer  program 
configuration  item  will  perform  its 
intended  function  for  a  specified 
interval  under  stated  conditions. 


Specification  A  document  intended  primarily  for  use 

in  procurement,  which  clearly  and 
accurately  describes  the  essential 
technical  requirements  for  items,  materials 
or  services  including  the  procedures  by 
which  it  will  be  determined  that  the 
requirements  have  been  met. 

Specification  Addendun  This  data  item  is  used  to:  create  a  new 

computer  program  configuration  item 
which  is  similar  to  an  existing  conr>uter 
program  configuration  item,  with  minimum 
redesign  effort,  and  apply  a  formal 
means  of  writing  a  specification  for  a 
new  computer  program  configuration  item 
by  changing  an  existing  specificatio  for 
a  computer  program  configuration  item  in 
a  manner  which  permits  ready  comparison 
of  the  exact  relationship  between  two 
computer  program  configuration  items. 
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Specification  Change 
Notice  (SOI) 


System  Specification 


Development  Specification 


Product  Specification 


ftmction  (Performance) 


Fabrication  (Design) 


This  document  is  used  to  propose,  trans¬ 
mit,  and  record  changes  to  a  specifica¬ 
tion.  In  the  proposal  or  preliminary 
form,  prior  to  approval,  the  SOI  supplies 
copies  of  the  pages  containing  the 
proposed  changes. 

A  document  which  states  the  technical 
and  mission  requirements  for  a  system  as 
an  entity,  allocates  requirements  to 
functional  areas  (or  configuration 
items) ,  and  defines  the  interfaces  be¬ 
tween  or  among  the  functional  areas. 

A  document  applicable  to  an  item  below 
the  system  level  which  states  perform¬ 
ance,  interface,  and  other  technical 
requirements  in  sufficient  detail  to 
pennit  design,  engineering  for  service 
use,  and  evaluation. 

A  document  applicable  to  a  production 
item  below  the  system  level  which  states 
item  characteristics  in  a  manner  suitable 
for  procurement,  production,  and  accept¬ 
ance. 

A  product  specification  which  states: 

(1)  the  conplete  performance  require¬ 
ments  of  the  product  for  the  intended 
use,  and  (2)  the  necessary  interface 
and  interchangeability  characteristics. 

It  covers  form,  fit,  and  function. 

A  product  specification  which  states: 

(1)  a  detailed  description  of  the  parts 
and  assemblies  of  the  product,  usually 
by  prescribing  compliance  with  a  set 
of  drawings,  and  (2)  those  performance 
requirements  and  corresponding  tests 
and  inspections  necessary  to  assure 
proper  fabrication,  adjustment,  and 
assembly  techniques. 
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System  Design  Review  (SDR) 


System  Requirements 


Subcontractor 


System 


This  review  shall  be  conducted  when  the 
definition  effort  has  proceeded  to  the 
point  where  system  requirements  and  the 
design  approach  are  more  precisely 
defined,  (i.e.,  alternate  design 
approaches  and  corresponding  test 
requirements  have  been  considered  and 
the  contractor  has  defined  and  selected 
the  required  equipment,  logistic  support, 
personnel,  procedural  data,  and  facili- 
ties).  This  review  shall  be  in  suffi¬ 
cient  detail  to  ensure  a  technical 
understanding  between  the  contractor  and 
the  procuring  activity  on:  the  system 
segments  identified  in  the  system 
specification  and  the  configuration 
item  identified  in  the  configuration 
item  performance  specification(s) . 

(MIL- STD- 1521) 

Review  (SRR)  The  objective  of  this  review  is  to 

ascertain  the  adequacy  of  the  contractor's 
efforts  in  defining  system  requirements. 

It  will  be  conducted  when  a  significant 
portion  of  the  system  functional  require¬ 
ments  has  been  established.  (MIL- STD- 1521) 

A  "subcontractor"  is  an  individual, 
partnership,  corporation,  or  association, 
who  (which)  contracts  with  a  contractor 
to  design,  develop,  design  and  manu¬ 
facture,  manufacture  items,  which  are  or 
were,  designed  specifically  for  use  in 
military  application. 

A  composite  of  subsystems,  assemblies 
(or  sets),  skills,  and  techniques 
capable  of  performing  and/or  supporting 
an  operational  (or  non-operational) 
role.  A  complete  system  includes 
related  facilities,  items,  material, 
services,  and  personnel  required  for 
its  operation  to  the  degree  that  it  can 
be  considered  a  self-sufficient  item  in 
its  intended  operational  (or  non- 
operational)  and/or  support  environment. 
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9ystflm/Subsystem  Specification 


Test  Plan 


Test  Report 


Vendor 

Version  Description  Document 


This  specification  may  be  prepared  to 
guide  development  of  large  projects  and 
■ay  be  used  to  prepare  individual  sub¬ 
system  specifications.  The  system/ 
subsystem  is  a  technical  document  pre¬ 
pared  to  describe  the  system.  It  is 
detailed  as  much  as  possible  concerning 
environment  and  design  to  provide 
maximum  guidance  for  program  design. 

All  system/subsystem  interfaces  are 
defined.  (SEOiAVINST  5233.1) 

The  acceptance  test  plan,  provdes  an 
overall  integrated  outline  of  the  total 
test  program,  program,  including:  test 
objectives,  identification  of  test  areas, 
and  responsibilities. 

A  document  containing  the  results  and 
analyses  of  tests  executed  during 
validation  and  acceptance  testing. 

A  ’Vendor"  is  a  manufacturer  or 
supplier  of  a  commercial  item. 

This  data  item  shall  be  used  to  accompany 
changes  to  an  approved  and  released 
computer  program.  Its  purpose  is  to 
identify  the  changes  made  and  the  exact 
version  of  a  computer  program  to  be 
delivered. 


Waiver  A  written  authorization  to  accept  a 

configuration  item  or  other  designated 
items,  which  during  production  ot  after 
having  been  submitted  for  inspection, 
are  found  to  depart  from  specified 
requirements,  but  nevertheless  are 
considered  suitable  for  use  "as  is"  or 
after  rework  by  an  approved  method. 
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APPENDIX  H 

GLOSSARY  FOR  SQA  TOOLS  AND  TECHNIQUES 


TECHNIQUES 


The  Twenty-One  Techniques  are  Listed 
in  Tables  X,  XI,  and  XII 


1.  ALGORITHM  EVALUATION  TEST.  A  technique  used  to 
evaluate  critical  algorithm  trade-offs  (i.e.,  speed  versus 
size  versus  precision)  before  the  design  is  finalized. 
Often  called  "the  hardest  out  first  method, "  the  technique 
creates  a  detailed  design  based  upon  trial  coding  results 
for  key  algorithms.  The  algorithms  are  often  extensively 
exercised  in  a  simulated  environment  to  ensure  mission 
requirements  are  satisfied. 

2.  ANALYTICAL  MODELING.  A  technique  used  to  express 
mathematically  (usually  by  a  set  of  equations)  a 
representation  of  some  real  problem.  Such  models  are 
valuable  for  abstracting  the  essence  of  the  subject  of 
inquiry.  Because  equations  describing  complex  systems  tend 
to  be  complicated  and  often  impossible  to  formulate,  it  is 
often  necessary  to  make  simplifying  assumptions,  which  may 
tend  to  distort  accuracy  (Ref  90). 

3.  AUDITING.  A  formal  technique  employed  to  examine  and 
verify  through  inspection  either  the  status  of  a  program 
and  its  documentation  or  the  adherence  of  project  personnel 
to  established  procedures.  Scheduled  audits  are  normally 
contractually  imposed  and  periodically  held.  Unscheduled 
audits  are  utilized  at  random  intervals  to  assess 
compliance  with  quality  requirements. 

4.  CODE  INSPECTION.  A  disciplined  technique  used  for 
inspecting  the  code  and  identifying  errors.  Participants 
have  well-defined  roles  and  criteria  for  evaluating  the 
code.  If  errors  are  identified,  the  code  is  reworked. 
Follow-up  procedures  are  used  to  ensure  that  the  errors 
have  been  corrected  (Ref  29). 

5.  CORRECTNESS  PROOFS.  A  technique  used  to  prove  the 
correctness  of  programs  using  means  similar  to  those 
employed  to  prove  mathematical  theorems.  Axioms  and 
theorems  derived  are  used  to  establish  the  validity  of  the 
program  with  respect  to  a  precise  specification  of  its 
purpose.  The  most  frequently  used  method  is  known  as  the 
inductive  assertion  or  Floyd  method  (Ref  91).  Several 
approaches  are  being  pursued.  One  approach  seeks  to 
demonstrate  program  correctness  a  priori  by  establishing 
the  proof  prior  to  implementation.  Another  approach  uses 
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an  interactive  system  to  prove  correctness  a 
posteriori  (Ref  92). 

6.  DESIGN  INSPECTION.  A  disciplined  technique  used  for 
inspecting  the  design  and  identifying  errors.  Participants 
have  well-defined  roles  and  criteria  for  evaluating  the 
design.  If  errors  are  identified,  the  design  is  reworked. 
Follow-up  procedures  are  used  to  ensure  that  the  errors 
have  been  corrected  (Ref  29). 

7.  ERROR-PRONE  ANALYSIS.  A  technique  employed  during 
coding  to  identify  areas  of  the  program  that  have  required 
abnormally  frequent  correction  and  change.  These  areas  can 
either  be  reworked  or  subjected  to  an  extensive  test 
effort  (Ref  93). 

8.  EQUIVALENCE  CLASSES.  A  technique  used  to  automatically 
identify  a  complete  set  of  test  cases  for  a  program.  The 
set  is  interpreted  in  terms  of  inequalities  involving 
program  variables  that  define  a  set  of  conditions  necessary 
for  the  particular  program  flow  to  actually  occur.  Some 
experience  in  the  practical  application  of  the  technique 
has  been  reported  (Refs  94,  95). 

9.  EXECUTION  ANALYSIS.  A  technique  employed  during  test 
to  investigate  program  behavior  errors  and  to  identify 
areas  in  the  code  that  were  either  untested  or  not  fully 
tested.  The  program  is  executed  and  statistics  are 
collected.  Test  results  and  the  statistics  are  then 
analyzed  to  insure  that  each  interface,  functional  and  test 
requirement  has  been  correctly  mechanized  by  the  code. 

10.  FUNCTIONAL  TESTING.  A  technique  used  to  demonstrate 
that  the  software  performs  its  specifications 
satisfactorily  under  normal  operating  conditions,  computing 
nominally  correct  output  values  from  nominal  input 
values  (Ref  96). 

11.  LOGICAL  TESTING.  A  technique  used  to  confirm  that  the 
code  performs  its  computation  correctly.  Items  validated 
oy  logical  testing  include  arithmetic  (i.e.,  precision, 
accuracy,  etc. ),  error  handling,  initialization, 
interfaces,  and  timing  (Ref  96). 

12.  PATH  TESTING.  A  technique  used  to  confirm  that  certain 
test-effectiveness  measures  based  on  the  program's  control 
topology  have  been  realized.  The  technique  assures  that  a 
sufficient  number  of  statements,  branch  paths,  and 
subroutine  calls  have  been  exercised  during  the  program 
execution.  It  also  helps  identify  a  complete  set  of  test 
cases  for  the  program  (Ref  97). 


13.  POST-FUNCTIONAL  ANALYSIS.  A  technique  employed  after 
completion  of  functional  testing  to  identify  functionally 
weak  areas  in  the  program.  The  recorded  test  results  are 


analyzed  and  the  quality  of  the  final  product  is 
determined  (Ref  93). 

14.  REVIEWING.  A  technique  employed  to  examine  and  verify 
through  inspection  either  the  status  of  a  program  and  its 
documentation  or  adherence  of  project  personnel  to 
established  procedures.  Scheduled  reviews  are  normally 
contractually  imposed  and  periodically  held  (Ref  87). 
Informal  reviews  are  held  frequently  to  assess  in  detail 
the  technical  adequacy  of  the  software  product  (Ref  98). 

15.  SIMULATION.  Simulation  is  the  process  of  studying 
specific  system  characteristics  by  use  of  models  exercised 
over  a  period  of  time  and  a  variety  of  conditions  for  the 
purpose  of  evaluating  alternatives,  timing,  system 
capacities,  performance,  and  constraints  within  the 
confines  of  that  system  (Ref  99).  Simulation  can  be  used 
by  quality  assurance  throughout  the  life  cycle.  It  can 
assist  in  evaluating  conceptual  trade-offs  (Ref  100).  It 
can  also  be  used  to  model  the  environment  and  provide 
realistic  test  inputs  to  a  program  being  examined. 

16.  SOFTWARE  QUALITY  METRICS.  A  technique  used  to 
quantitatively  assess  the  quality  of  software  throughout 
the  SDLC.  Measurements  or  metrics  provide  a  software 
monitoring  method  which  gives  an  indication  of  the 
progression  toward  a  specified  quality  (Ref  17). 

17.  STANDARDIZATION.  A  technique  used  to  create  an 
authoritative  model  against  which  products  and/or 
procedures  can  be  compared  in  order  to  determine  their 
quality.  Software  items  for  which  standards  can  be  easily 
established  include  documentation  (Refs  101,  102), 
languages  (Refs  103,  104),  designs  (Refs  105,  8,  106),  and 
structured  programming  (Ref  107). 

18.  STATIC  ANALYSIS.  A  technique  employed  during  test  to 
identify  weaknesses  in  the  source  code.  The  syntax  of  a 
program  is  examined  and  statistics  about  it  are  generated. 
Items  such  as  relationships  between  module,  program 
structure,  error-prone  constructions,  and  symbol/subroutine 
cross-references  are  checked  and  violations  of  established 
rules  are  analyzed. 

19.  STRESS  TESTING.  A  technique  employed  to  confirm  that 
the  code  performs  its  specifications  satisfactorily  under 
extreme  operating  conditions,  computing  nominally  correct 
output  values  from  worst  case  input  values  (i.e., 
singularities,  end  points  for  the  range  of  data,  etc.). 

20.  SYMBOLIC  EXECUTION.  A  technique  that  employs  symbolic 
data  to  confirm  that  the  software  performs  properly. 
Symbolic  execution  al'.ows  one  to  choose  intermediate  points 
in  the  test  spectrum  ranging  from  individual  test  runs  to 
correctness  proofs.  Its  results  can  be  used  to  develop  a 


28  6 


minimum  set  of  test  cases  (Refs  108,  109). 


21.  WALK-THROUGHS.  A  technique  used  for  reviewing  the 
design  or  code  and  identifying  errors.  The  responsible 
programmer  discusses  his  product  with  his  peers  and 
solicits  their  constructive  advice.  Product  modifications 
are  then  made  at  the  discretion  of  the  programmer  to 
correct  problems  identified  during  the  review  (Ref  110). 
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TOOLS 


The  Thirty-Three  Tools  are  Listed 
in  Tables  X,  XIII,  and  XIV 


1.  ACCURACY  STUDY  PROCESSOR.  A  computer  program  used  to 
perform  calculations  or  assist  in  determining  if  program 
variables  are  computed  with  required  accuracy.  The 
processor  accepts  mathematical  equations  and  data  as 
inputs.  It  then  uses  the  data  as  variables  in  the 
equations  and  solves  them  (Ref  111). 

2.  AUTOMATED  MEASUREMENT  TOOL.  A  computer  software 
package  that  collects  development  data  through  manual  and 
automatic  input.  The  AMT  derives  metric  scores  which 
reflect  the  level  of  quality  that  the  developing  software 
exhibits.  It  generates  reports  that  show  a  quantitative 
measure  of  the  quality  attributes  and  factors  being 
considered  (Refs  17,  23,  88). 

3.  AUTOMATED  TEST  GENERATOR.  A  computer  program  that 
accepts  inputs  specifying  a  test  scenario  in  some  special 
language,  generates  the  exact  computer  inputs,  and 
determines  the  expected  results.  The  NASA -developed 
Automated  Test  Data  Generator  ( ATDG )  serves  as  one 
example  (Ref  112).  ATDG  takes  identified  code  segments, 
determines  all  possible  logic  transfers  between  those 
segments,  defines  a  path  through  the  module  under  test,  and 
generates  input  data  required  to  execute  the  selected 
paths.  The  General  Electric  automated  software  test  driver 
serves  as  Mother  example  (Ref  113).  This  program 
automates  the  process  of  test  planning,  test  setup,  test 
execution,  and  test  analysis.  Other  generators  are 
described  in  a  recent  report  (Ref  114). 

4.  COMPARATOR.  A  computer  program  used  to  compare  two 
versions  of  the  same  computer  program  under  test  to 
establish  identical  configuration  or  to  specifically 
identify  changes  in  the  source  coding  between  the  two 
versions  (Ref  90). 

5.  CONSISTENCY  CHECKER.  A  computer  program  used  to 
determine  (1)  if  requirements  and/or  designs  specified  for 
computer  programs  are  consistent  with  each  other  and  their 
data  base  and  (2)  if  they  are  complete.  The  consistency 
checker  computer  program  developed  by  TRW  is  an 
example  (Ref  115). 
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6.  CROSS-REFERENCE.  A  group  of  computer  programs  that 
provide  cross-reference  information  on  system  components. 
For  example,  programs  can  be  cross-referenced  with  other 
programs,  macros,  parameter  names,  etc.  This  capability  is 
useful  in  problem-solving  and  testing  to  assess  impact  of 
changes  to  one  area  or  another  (Ref  90).  This  capability 
should  be  provided  in  most  compiler  environments  (Ref  116). 

7.  DATA  BASE  ANALYZER.  A  computer  program  that  reports 
information  on  every  usage  of  data,  identifies  each  program 
using  any  data  elements,  and  indicates  whether  the  program 
inputs,  uses,  modifies,  or  outputs  the  data  element.  Any 
unused  data  is  printed.  Errors  dealing  with  misuse  and 
nonuse  of  data  and  conflicts  in  data  usage  are  identified 
during  the  analysis  (Ref  90). 

8.  DEBUGGER.  Compile  and  execution-time  check-out  and 
debug  capabilities  that  help  identify  and  isolate  program 
errors.  They  usually  include  commands  or  directives  such 
as  DUMP,  TRACE,  MODIFY  CONTENTS,  BREAKPOINT,  etc.  (Ref  90). 
Some  debuggers  operate  at  the  source  level  (e.g.,  SIMDDT 
for  SIMULA  67,  and  COBDDT  for  COBOL  on  the  PDP-10 )  and 
others  at  the  objective  level  with  some  additional  source 
information  (e.g.,  DDT  for  FORTRAN  on  the  PDP-10)  (Ref  117). 
The  following  types  of  basic  information  are  normally 
provided : 

TRACE:  A  timed  record  of  program  execution  and 

machine  environment  information. 

DUMP:  A  record  of  selected  portions  of  memory  or 

register  usage  output  after  a  specified  point  in  the 
program's  execution  has  been  reached. 

SNAP:  A  record  of  intermediate  values  of  items 
captured  during  program  execution. 

BREAKPOINTS:  A  facility  whereby  the  normal  computation 

is  interrupted  and  debugging  activities  commence 
execution. 

9.  DECISION  TABLES.  A  mechanism  used  to  represent 
information  on  program  conditions,  rules,  and  actions  in 
tabular  form  that  can  be  automatically  translated  to 
executable  code  by  a  processor.  Decision  tables  are  a 
tabular  representation  of  the  design  which  can  be  used  to 
clarify  the  control  flow  of  decision  alternatives  by 
presenting  the  information  in  a  concise  and  understandable 
format  (Ref  99).  A  dated  but  useful  survey  ot  automated 
decision  table  processors  is  available  (Ref  118). 

10.  DYNAMIC  ANALYZER.  A  computer  program  that  instruments 
the  source  code  by  adding  counters  and  other 


statistics-gathering  sensors  and  produces  reports  on  how 
thoroughly  the  various  portions  of  the  code  have  been 
exercised  after  the  augmented  code  is  executed.  Dynamic 
analyzers  provide  information  useful  for  tuning, 
optimization,  and  test  case  design  (Ref  90).  Many  dynamic 
analyzers  have  been  developed.  Comparative  analysis  of 
several  analyzers  (Ref  119)  and  user  feedback  with  such 
systems  have  been  published  (Ref  120). 

11.  DYNAMIC  SIMULATOR.  A  computer  program  used  to  check 
out  a  program  in  a  simulated  environment  comparable  to  that 
in  which  it  will  reside.  Closed-loop  effects  between 
computer  and  environmental  models  are  gained  when  the 
various  models  respond  to  inputs  and  outputs.  The 
simulator  allows  the  environment  to  be  stabilized  at  a 
specific  configuration  for  any  number  of  runs  required  to 
observe,  diagnose,  and  resolve  problems  in  the  operational 
program  (Ref  90).  The  Dynamic  Test  Station  (DTS)  being 
developed  by  the  U. S.  Air  Force  to  support  the  test  of  the 
operational  flight  software  for  the  F-16  fighter  aircraft 
serves  as  an  example  (Ref  121). 

12.  EDITOR.  A  computer  program  used  to  analyze  source 
programs  for  coding  errors  and  to  extract  information  that 
can  be  used  for  checking  relationships  between  sections  of 
code.  The  editor  can  scan  source  code  and  detect 
violations  to  specific  programming  practices  and  standards, 
construct  an  extensive  cross-reference  list  of  all  labels, 
variables  and  constants,  and  check  for  prescribed 
program  formats  (Ref  90). 

13.  FLOWCHARTER.  A  computer  program  used  to  show  in  detail 
the  logical  structure  of  a  computer  program.  The  flow  is 
determined  from  the  actual  operations  as  specified  by  the 
executable  instructions,  not  from  comments.  The  flowcharts 
generated  can  be  compared  to  flowcharts  provided  in  the 
computer  program  design  specification  to  show  discrepancies 
and  illuminate  differences  (Ref  90).  Several  flowcharters 
such  as  AUTOFLOW  and  FLOWGEN  are  commercially  available. 
Several  structured  flowchart  representations  have  been 
proposed  (Ref  122). 

14.  HARDWARE  MONITOR.  A  unit  that  obtains  signals  from  a 
host  computer  through  probes  attached  directly  to  the 
computer's  circuitry.  The  signals  obtained  are  fed  to 
counters  and  timers  and  are  recorded.  These  data  are  then 
reduced  to  provide  information  about  system  and/or  program 
performance  (CPU  activity,  channel  utilization, 
etc.)  (Ref  123).  Several  useful  articles  on  using  this 
tool  have  appeared  in  the  literature  (Refs  124,  125). 

15.  INSTRUCTION  TRACE.  A  computer  program  used  to  record 
every  instance  a  certain  class  of  operations  occurs  and 
triggers  event-driven  data  collection.  In  some  cases,  this 
creates  a  complete  timed  record  of  events  occurring  during 
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program  execution  (Ref  90).  Experience  using  traces  to 
locate  sources  of  nonrepeatable,  intermittent  malfunctions 
has  been  reported  (Ref  126). 

16.  INTERFACE  CHECKER.  A  computer  program  used  to 
automatically  check  the  range  and  limits  of  variables  as 
well  as  the  scaling  of  the  source  program  to  assure 
compliance  with  interface  control  documents.  Other 
computer  programs  have  been  developed  to  automatically 
verify  that  modularity  rules  have  been  followed  (Ref  127). 

17.  INTERRUPT  ANALYZER.  A  computer  program  that  determines 
potential  conflicts  to  a  system  as  a  result  of  the 
occurrence  of  an  interrupt  (Ref  90). 

18.  LANGUAGE  PROCESSORS.  Computer  programs  used  to 
translate  high-level  or  symbolic  instruction  mnemonics  into 
computer-oriented  code  capable  of  being  executed  by  a 
computer.  Compilers,  assemblers  and  me ta -assemblers  are 
example  tools  used  for  program  development.  Other  language 
processors  have  been  developed  to  support  requirements 
generation  and  validation  (Ref  128),  design  (Ref  129),  and 
test  (Ref  113).  Preprocessors  have  been  developed  to 
support  implementation  of  modern  programming  techniques. 

19.  LIBRARIES.  A  collection  of  organized  information  used 
for  reference  or  study.  Many  varieties  of  library  systems 
can  be  implemented.  Some  manage  the  storage  and 
distribution  of  the  computer  program  in  both  source  and 
object  form.  Others  manage  the  computer  program,  its 
documentation  and  related  test  data  (i.e.,  test  cases, 
procedures,  results).  Programs  which  support  their 
implementation  are  commercially  available  and  include 
Applied  Data  Research's  LIBRARIAN  and  International 
Business  Machine's  Program  Production  Library  (PPL) 
(Ref  90).  The  use  of  a  library  and  its  effect  on 
productivity  have  been  reported  (Ref  130). 

20.  LOGIC  ANALYZER.  A  computer  program  used  to 
automatically  reconstruct  equations  forming  the  basis  of  a 
program  and  to  flowchart  assembly  language  programs.  One 
such  program  translates  assembly  language  instructions  into 
a  machine-independent  microprogramming  language  and  builds 
the  microprogramming  statements  into  a  network  in  which  the 
flow  of  control  is  analyzed  and  equations  are 
reconstructed  (Ref  131). 

21.  MANAGEMENT  INFORMATION  SYSTEM.  Consists  of  a 
computer-based  information  system  (a  particular  combination 
of  human  service,  material  service,  and  equipment  service) 
for  the  purpose  of  gathering,  organizing,  communicating, 
and  presenting  information  to  be  used  by  individuals  for 
planning  and  controlling  an  enterprise  (Ref  132).  Several 
packages  which  serve  as  essential  elements  of  a  management 
information  system  are  discussed  with  examples  in  a  recent 
publication  (Ref  133). 
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22.  REQUIREMENTS  TRACER.  A  computer  program  used  to 
provide  traceability  from  requirements  through  design  and 
iag>lementation  of  the  software  products.  Traceability  is 
characterized  to  the  extent  that  an  audit  trail  exists  for 
the  successive  implementation  of  each  requirement.  The 
University  of  Michigan-developed  Problem  Statement 
Language/Analyzer  serves  as  an  example  (Ref  134). 
Experience  using  the  Michigan  and  other  similar  systems  has 
been  reported  (Ref  135). 

23.  SIMULATOR.  A  computer  program  that  provides  the  target 
system  with  inputs  or  responses  that  resemble  those  that 
would  have  been  provided  by  the  process  for  the  device 
being  simulated.  The  simulator's  function  is  to  present 
data  to  the  system  at  the  correct  time  and  in  an  acceptable 
format  (Ref  90).  Several  of  the  many  varieties  of 
simulators  available  are  briefly  described  as  follows: 

Interpretive  computer  simulator:  A  computer  program 
used  to  simulate  the  execution  characteristics  of  a 
target  computer  (provides  bit-for-bit  fidelity  with 
results  that  would  be  produced  by  the  target  machine) 
using  a  sequence  of  instructions  of  the  host  computer. 

Peripheral  simulator:  A  computer  program  used  to 
present  functional  and  signal  interfaces  representative 
of  a  peripheral  device  to  the  target  system. 

Statement-level  simulator:  A  computer  program  used  to 
simulate  the  execution  characteristics  of  a  target 
computer  at  the  source  instruction-level  using  a 
sequence  of  instructions  on  the  host  computer. 

System  simulator:  A  mechanization  of  a  model  of  the 
system  (hardware,  software,  interfaces)  used  to  predict 
system  performance  over  time. 

24.  SOFTWARE  MONITOR.  A  computer  program  that  provides 
detailed  statistics  about  system  performance.  Because 
software  monitors  reside  in  memory,  they  have  access  to  all 
the  tables  the  system  maintains.  Therefore,  they  can 
examine  such  things  as  core  usage,  queue  lengths,  and 
individual  program  operation  to  help  measure  performance. 
Use  of  software  monitors  has  been  described  in  a  recent 
publication  (Ref  136). 

25.  STANDARDS.  Procedures,  rules,  and  conventions  used  for 
prescribing  disciplined  program  development.  Architecture 
and  partitioning  rules,  documentation  conventions,  language 
conventions,  configuration,  and  data  management  procedures, 
etc.,  are  typical  examples  under  this  category  (Ref  123). 

26.  STANDARDS  ANALYZER.  A  computer  program  used  to 
automatically  determine  whether  prescribed  programming 
standards  and  practices  have  been  followed.  The  program 
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can  check  for  violations  to  standards  set  for  such 
conventions  as  program  size,  commentary,  structure, 
etc.  (Ref  90).  The  National  Bureau  of  Standards -developed 
FORTRAN  Analyzer  serves  as  an  example  (Ref  137). 

27.  STATIC  ANALYZER.  A  computer  program  used  to  provide 
information  about  the  features  of  a  source  program.  This 
type  of  tool  examines  the  source  code  statically  (not  under 
execution  conditions)  and  performs  syntax  analysis, 
structure  checks,  module  interface  checks,  event  sequence 
analysis  and  other  similar  functions.  Several  of  the  many 
varieties  of  static  analyzers  available  are  briefly 
described  as  follows: 

Overlay  analyzer:  A  computer  program  that  examines  the 
source  program  in  order  to  determine  mutually  disjoint 
segments  that  can  reside  in  the  same  area  of  memory  at 
run  time  (Ref  117). 

Units  consistency  analyzer:  A  computer  program  which 
analyzes  the  source  code  version  of  equations  to  assure 
that  they  consistently  reference  the  global  data 
base  (Ref  137). 

Usage  statistics  gatherer:  A  computer  program  that 
computes  statistics  based  on  the  number  of  times 
various  items  appear  in  a  source  program  (Ref  117). 

28.  STRUCTURE  ANALYZER.  A  computer  program  used  to  examine 
source  code  and  determine  that  structuring  rules  set  for 
either  the  control  or  data  structure,  or  both,  have  been 
obeyed.  Typically,  the  program  parses  an  equivalent  model 
of  the  control  or  data  topology  before  commencing  analysis. 

29.  TEST  BED.  A  test  site  composed  of  actual  hardware 
(hardware  test  site)  or  simulated  equipment  (software  test 
site)  or  some  combination.  A  hardware  test  site  uses  the 
actual  computer  and  interface  hardware  to  check  out  the 
hardware/software  interfaces  and  actual  input/output.  The 
program  execution  is  confirmed  using  actual  hardware  timing 
characteristics,  but  the  output  is  limited  and  test 
repeatability  is  a  problem.  A  software  test  site  uses  an 
instruction-level  and/or  statement-level  simulator  to  model 
actual  hardware.  A  software  test  site  permits  full  control 
of  inputs  and  computer  characteristics,  allows  processing 
of  intermediate  outputs  without  destroying  simulated  time, 
and  allows  full  test  repeatability  and  good  diagnostics. 
The  Shuttle  Avionics  Integration  Laboratory  represents  an 
elaborate  hardware  test  bed  (Ref  138)  and  the  Software 
Design  and  Verification  System  (SDVS)  represents  a 
sophisticated  software  test  bed  (Ref  139). 

30.  TEST  DRIVERS,  SCRIPTS.  To  run  tests  in  a  controlled 
manner,  it  is  often  necessary  to  work  within  the  framework 
of  a  "scenar io"--a  description  of  a  dynamic  situation.  To 
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accomplish  this,  the  input  data  files  for  the  system  must 
be  loaded  with  data  values  representing  the  test  situation 
or  events  to  yield  recorded  data  to  evaluate  against 
expected  results.  These  tools  permit  generation  of  data  in 
external  form  to  be  entered  into  the  system  at  the  proper 
time  (Ref  90). 

31.  TEST-RESULT  PROCESSOR.  A  computer  program  used  to 
perform  test  output  data  reduction,  formatting,  and 
printing.  Some  perform  statistical  analysis  where  the 
original  data  may  be  the  output  of  a  monitor  (Ref  90). 

32.  TEXT  EDITOR.  A  computer  program  used  to  prepare 
documentation  and  perform  work-file  edits  (erase,  insert, 
change,  and  move  words  or  groups  of  words).  The  program 
requires  a  facility  for  on-line  storage  and  recall  of  text 
units  for  inspection,  editing,  or  printing  (Ref  90). 

33.  TIMING  ANALYZER.  A  computer  program  that  monitors  and 
prints  execution  time  for  all  program  elements  (functions, 
routines,  and  subroutines).  A  more  detailed  description  ol 
the  tool  appears  in  the  literature  (Ref  131). 
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A  survey  of  software  tools  on  the  candidate  target 
environments  was  conducted  during  the  first  phase  of  the 
AMT  development.  84  The  purpose  of  the  survey  was  to 
identify  software  tools  that  could  be  incorporated  in  the 
AMT.  The  survey  was  limited  to  the  candidate  environments 
because  it  was  felt  it  was  beyond  the  scope  of  this  effort 
to  transport  tools  from  other  environments.  The  criteria 
for  selection  of  a  tool  for  consideration  for  incorporation 
in  the  AMT  were: 

o  Applicability  to  software  measurement  (Did  the  tool 
provide  any  metric  data? ) 

o  Portability  of  tool  (Can  the  tool  be  used  on  differ¬ 
ent  hardware  configurations?) 

o  Interoperability  of  the  tool  (How  many  modifications 
to  the  tool  are  necessary?  ) 

o  Usability  of  the  tool  (How  much  effort  is  required 
to  learn  how  to  operate  the  tool?  How  much  effort 
is  there  to  preparing  input  and  interpreting  output 
that  was  tool-driven?) 

The  results  of  this  Tool  Survey  are  presented  in  matrix 
form.  Background  information  and  analysis  of  the 
state-of-the-art  of  software  tools  and  their  applicability 
to  metric  appear  before  it,  preceding  the  Tools  Survey.  As 
a  result  of  this  analysis,  selective  tools  that  have 
compatible  hardware/operating  systems  with  the  target 
environments  are  also  included  in  the  matrix.  Finally,  the 
last  part  deals  with  the  actual  tools  to  be  used  in  the  AMT 
were  considered  for  use,  or  were  applied  during  its 
development. 


CODING  AND  IMPLEMENTATION:  METRICS  APPLICABILITY 

The  origin  of  code  inspection  was  structured  programming 
and  allied  software  engineering  technologies  of  the  early 
1970's.  The  goal  of  automated  static  analysis/evaluation 
has  been  to  automate  the  compliance  with  the  techniques  and 
make  a  search  of  program  properties. 
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The  program  parameters  are  structure-based  (program  logical 
and  data  structure,  naming  conventions,  documentation 
conventions,  etc.),  control/data  flow  based  (avoidance  of 
undue  control  complexity;  assurance  of  well-definedness  of 
variables,  etc.),  and  interface  based  (assurance  of 
correspondence  between  modules,  subsystem,  inter-system, 
etc.  ).  The  anomaly-detecting  metrics  have  to  do  with 
standards  enforcement  (deficiencies  in  source  code), 
whereas  the  predictive  metrics  quantify  the  logic  of  design 
and  implementation. 

For  exanqple,  the  JOVIAL  Automated  Metric  System  (JAMS)  is 
designed  to  collect  structural  information  about  JOVIAL 
programs.  GE's  Integrated  Software  Development  System 
( ISDS )  provides  a  capability  to  analyze  other  languages 
including  FORTRAN,  PDL,  IFTRAN ,  and  PASCAL.  A  major 
subsystem  of  ISDS,  the  generalized  parser  (GNP),  the 
grammar  description  language  (GDL )  and  grammar  tables, 
provides  this  capability  and  will  be  used  in  the  AMT. 

Symbolic  evaluation  of  code  has  as  its  goal  the 
"interpretation"  of  program  behavior  at  the  programming 
language  level.  Assunqption  must  be  made  about  the 
environment,  the  deterministic  properties  of  the 

programming  language  behavior,  and  the  outcome  of  symbolic 
execution  results.  On  systems  such  as  DISSECT  or  MACSYMA 
the  user  interactively  chooses  a  path  and  performs  symbolic 
interpretation  of  actions  along  the  chosen  path.  The 
system  then  displays  the  "formulas"  to  the  user.  The  user 
compares  original  and  inqalemented  formulas  for  equality. 
Differences  between  computed  and  actual  formulas  are 
mistakes.  Special  formula  formatting  methods  are  used  to 
make  these  differences  highly  visible.  Final  control 
software  is  not  yet  available.  symbolic  evaluation  has 
good  candidate  potential  for  the  accuracy  metrics  at  the 
system  level. 

The  final  type  of  static  analysis  tools,  proof  of 
correctness,  can  be  used  at  the  system  level,  subsystem 
level,  or  the  module  level  as  assessments  of  different 
levels  of  correctness.  The  Failure  of  Proof  Method  ( FPM ) , 
uses  a  mathematical  approach  to  proving  the  correspondence 
between  a  program  and  its  formal  specification.  The 
consistency  metric  is  highly  visible  here. 

Dynamic  testing  is  achieved  through  system  exercising  of 
programs.  Typical  self-testing  metrics  for  higher  level 
language  systems  have  been  built  on  a  experimental  basis 
and  include: 

o  Automatic  specified  percentage  of  program  logical 
segment  coverage  in  any  one  test;  aggregated  test 
coverage  of  close  to  100%. 
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o  Assistance  in  setting  input  values  and  evaluating 
output  values. 

o  Some  form  of  automated  results  comparison. 

These  dynamic  test  tools  consist  of  two  basic  modules,  an 
instrumentation  module  and  an  analyzer  module.  The  source 
language  program  is  submitted  directly  to  the 
instrumentation  module.  Then  the  instrumentation  module 
accepts  the  source  program  of  the  module  under  test  and 
instruments  it  by  inserting  additional  statements  in  the 
form  of  counters  or  sensors.  The  instrumented  source  file 
is  compiled  and  executed.  At  this  point  an  analyzer  module 
produces  a  report  documenting  the  behavior  under  the  test 
during  its  execution. 

Typical  metric-like  data  reported  are: 
o  Max  and  min  values  of  variables. 

o  Number  and  percentage  of  subroutine  calls  executed, 
o  Measures  of  program  complexity, 
o  Statement  consistency  checks, 
o  Program  cross-references, 
o  Trace  capability, 
o  Flagging  of  non-ANSI  :ode. 
o  Logically  impossible  -  path  detection, 
o  Subroutine  argument/parameter  verification, 
o  Data  range  check, 
o  If  statement  trace, 
o  Branch  trace, 
o  Subroutine/statement  timing, 
o  Min/max  assignment  values, 
o  First/last  assignment  values, 
o  Min/max  DO  Loop  Control  Variable, 
o  Final  DO  Loop  index  value, 
o  Final  branch  values. 
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o  Statement,  path,  segment,  module  interface  or  flow 
execution  frequencies. 

o  Specific  data  associated  with  each  executable  source 
statement. 

o  Subroutine  retrace  capability,  complete  calling 
tree,  reverse  execution  capability. 

o  Performance  indices  for  modules  and  input  data. 

A  list  of  dynamic  tools  would  include:  JAVS,  CABS,  FA VS, 
RXVP,  FORTUNE,  CIP,  FORSAP,  FETE,  PROGFORT,  PROGTIME,  TPL, 
and  TAP. 

The  goal  of  mutation  analysis  is  to  show  that  small  changes 
in  program  are  discovered  by  test  data.  Conversely,  the 
test  data  must  be  strong  enough  to  catch  the  significant 
errors.  Relevance  to  error  detection  metrics  is  obvious. 

The  Pilot  Mutation  System  (PIMS)  has  been  applied  tc 
FORTRAN  and  COBOL  pilot  systems.  Magnitude  of  the  mutant 
error  is  classified  as: 

o  Program  does  not  compute. 

o  Program  computes  but  does  not  run  test  data. 

o  Program  compiles,  test  run  is  satisfactory,  and  the 
program  is  either  logically  equivalent  to  the  ori¬ 
ginal  or  test  data  is  not  good  enough. 

Reliability  analysis  is  still  in  its  infancy.  The  goal  is 
to  determine  whether  all  defects  have  been  reliably  removed 
by  tests.  Any  error  must  be  made  known  by  some  combination 
of  inputs.  Following  this  theoretical  approach  of 
examining  all  possible  input  combinations  is  prohibitive  in 
terms  of  cost  effectiveness  and  computer  time/capacity. 
The  Next  Error  Discovery  Predition  method  fails  because 
software  reliability  simply  does  not  follow  the  probability 
laws  of  hardware  reliability. 


The  matrix  of  software  tools  having  potential  metric 
applicability  follows.  It  includes  tools  currently  in  use 
or  planned  for  at  RADC  and  additional  non -R ADC  tools  also 
worthy  of  consideration  for  AMT  development  or  usage. 


Figure  17  (continued 
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DESCRIPTION  OF  TOOL 


ISDS  (6E)  Integrated  Software  Development  System 
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PROCEDURES  FOR  ASSESSING  SOFTWARE  QUALITY 


The  benefits  of  applying  the  software  quality  metrics  are  realized  when 
the  information  gained  from  their  application  is  analyzed.  The  analyses 
that  can  be  done  based  on  the  metric  data  are  described  in  the  subsequent 
paragraphs.  There  are  three  levels  at  which  analyses  can  be  performed. 
These  levels  are  related  to  the  level  of  detail  to  which  the  quality 
assurance  organization  wishes  to  go  in  order  to  arrive  at  a  quality  assess¬ 
ment  (Ref  28). 


Inspector's  Assessment 

The  first  level  at  which  an  assessment  can  be  made  relies  on  the  discipline 
and  consistency  introduced  by  the  application  of  the  worksheets.  An 
inspector,  using  the  worksheets,  asks  the  same  questions  and  takes  the 
same  counts  for  each  module's  source  code  or  design  document,  etc.  that  is 
reviewed.  Based  on  this  consistent  evaluation,  a  subjective  comparison 
of  products  can  be  made. 

Document  Inspector's  Assessment.  The  last  section  in  each  worksheet 
is  a  space  for  the  inspector  to  make  comments  on  the  quality  observed 
while  applying  the  worksheet.  Comments  should  indicate  an  overall  assessment 
as  well  as  point  out  particular  problem  areas  such  as  lack  of  comments, 
inefficiencies  in  implementation,  or  overly  complex  control  flow. 

Compile  Assessments  for  System  Review.  By  compiling  all  of  the 
inspector's  assessments  on  the  various  documents  and  source  code  available 
at  any  time  during  the  development,  deficiencies  can  be  identified. 


Sensitivity  Analysis 

The  second  level  of  detail  utilizes  experience  gained  through  the  applica¬ 
tion  of  metrics  and  the  accumulation  of  historical  information  to  take 
advantage  of  the  quantitative  nature  of  the  metrics.  The  values  of  the 

measurements  are  used  as  indicators  for  evaluation  of  the  progress  toward 

a  high  quality  product. 

At  appropriate  times  during  a  large-scale  development,  the  application  of 
the  worksheets  allows  calculation  of  the  metrics.  The  results  of  these 
calculations  is  a  matrix  of  measurements.  The  metrics  that  have  been 
established  to  date  are  at  two  levels  --  system  level  and  module  level. 

The  approach  to  be  described  is  applicable  to  both  levels  and  will  be 

described  in  relationship  to  the  module  level  metric. 
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A  n  by  k  matrix  of  measurements  results  from  the  applicaticn  of  the 
metrics  to  the  existing  products  of  the  development  (e.g.,  at  design,  the 
products  might  include  review  material,  design  specifications,  test  plans, 
etc.)  where  there  are  k  modules  and  n  module  level  measurements  applicable 
at  this  particular  time. 
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This  matrix  represents  a  profile  of  all  of  the  modules  in  the  system  with 
respect  to  a  number  of  characteristics  measured  by  the  metrics.  The 
analyses  that  can  be  performed  are  described  in  the  following  steps: 

Assess  Variation  of  Measurements.  Each  row  in  the  above  matrix 
represents  how  each  module  in  the  system  scored  with  respect  to  a  parti¬ 
cular  metric.  By  summing  all  the  values  and  calculating  the  average  and 
standard  deviation  for  that  metric,  each  individual  module's  score  can 
then  be  compared  with  the  average  and  standard  deviation.  Those  modules 
that  score  less  than  one  standard  deviation  from  the  average  should  be 
identified  for  further  examination.  These  calculations  are  illustrated 
below: 


k 

for  metric  i ;  Average  Score  =  A.  =  s  M. ./k 

1  j=l  1J 

k  2 

Standard  Deviation  =  j.  =  as  (M.  .-A. )  /k 

1  j=l  13  1 

Report  Module  j  if  M^<A.  - 


Assess  Low  System  Scores.  In  examining  a  particular  measure  across 
all  modules,  consistently  low  scores  may  exist.  It  nay  be  that  a  design 
or  implementation  technique  used  widely  by  the  development  team  was  the 
cause.  This  situation  identifies  the  need  for  a  new  standard  or  stricter 
enforcement  of  existing  standaid'-  to  improve  the  overall  development 
effort. 

Assess  Scores  Against  Thresholds.  As  experience  is  gained  with  the 
metrics  and  data  is  accumulated,  threshold  values  or  industry  acceptable 
limits  may  be  established.  The  scores,  for  each  module  for  a  particular 
metric  should  be  compared  with  the  established  threshold.  A  simple 
example  is  the  percent  of  comments  per  line  of  source  code.  Certainly 
code  which  exhibits  only  one  or  two  percent  measurements  for  this  metric 
would  be  identified  for  corrective  action.  It  may  be  that  ten  percent 
is  a  minimum  acceptable  level.  Another  example  is  the  complexity  measure. 
A  specific  value  of  the  complexity  measure  greater  than  some  chosen  value 
should  be  identified  for  corrective  action. 
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Report  Module  j  if  M.^cT.  (or>T  for  complexity  measures) 

Where  T..  =  threshold  value 
specified  for  metric  i 


Use  of  Normalization  Function  to  Assess  Quality 

The  last  level  of  assessing  Quality  is  using  the  normalization  functions  to 
predict  the  quality  in  quantitative  terms.  The  normalization  functions 
are  utilized  in  the  following  manner. 

For  a  particular  time  there  is  an  associated  matrix  of  coefficients  which 
represent  the  results  of  linear  multivariate  regression  analyses  against 
empirical  data  (past  software  developments).  These  coefficients,  when 
multiplied  by  the  measurement  matrix  results  in  an  evaluation  (prediction) 
of  the  quality  of  the  product  based  on  the  development  to  date.  This 
coefficient  matrix,  shown  below,  has  n  columns  for  the  coefficients  of 
the  various  metrics  and  11  rows  for  the  11  quality  factors. 

f  C11  c12  .  .  .  cln  1 


L  cll,l  C1 1  ,n  -I 


To  evaluate  the  current  degree  or  level  of  a  particular  quality  factor,  i, 
for  a  module,  j,  the  particular  column  in  the  measurement  matrix  is 
multiplied  by  the  row  in  the  coefficient  matrix.  The  resultant  value: 


Ci  ,1 


+  Ci,2 


2,j 


+  c  • 
i  ,n 


m,j 


r .  . 
i  ,J 


is  the  current  predicted  rating  of  that  module,  j .  for  the  quality  factor,  i 
This  predicted  rating  is  then  compared  to  the  previously  established  rating 
to  determine  if  the  quality  is  at  least  as  sufficient  as  required.  The 
coefficient  matrix  should  be  relatively  sparse  (many  Cn- j  -  0).  Only  sub¬ 
sets  of  the  entire  set  of  metrics  applicable  at  any  one  time  relate  to  the 
criteria  of  any  particular  quality  factor. 

Multiplying  the  complete  measurement  matrix  by  the  coefficient  matrix 
results  in  a  ratings  matrix.  This  matrix  contains  the  current  predicted 
ratings  for  each  module  for  each  quality  factor.  Each  module  then  can  be 
compared  with  the  preset  rating  for  each  quality  factor. 


CM=R^= 


rll  r12 


l.k 


L  '11,1 


11, k  J 


',10 
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This  approach  represents  the  most  formal  approach  to  evaluating  the 
quality  of  a  product  utilizing  the  software  quality  metrics. 

To  use  the  normalization  functions  that  currently  exist  the  following 
steps  should  be  performed. 

AppIv  Normalization  Function.  Table  XXI  contains  the  normal ization 
functions  that  currently  exist.  If  any  of  the  quality  factors  identified  in 
that  table  have  been  specified  as  a  requirement  of  the  development,  then 
the  metrics  identified  in  the  table  should  be  substituted  into  the  equation 
and  the  predicted  rating  calculated.  Normalization  functions  which 
include  several  metrics  can  be  used  if  available,  otherwise  functions  for 
individual  metrics  should  be  used.  This  predicted  rating  should  be  com¬ 
pared  with  the  specified  rating. 

To  illustrate  the  procedure  the  normalization  function  that  has 
been  developed  for  the  factor  Flexibility  will  be  used.  The  normalization 
function,  applicable  during  the  design  phase,  relates  measures  of  modular 
implementation  to  the  flexibility  of  the  software.  The  predicted  rating 
of  flexibility  is  in  terms  of  the  average  time  to  implement  a  change  in 
specifications.  The  normalization  function  is  shown  in  figure  19.  The 
measurements  associated  with  the  modular  implementation  metric  are  taken 
from  design  documents.  The  measurements  involve  identifying  if  input, 
output  and  processing  functions  are  mixed  in  the  same  module,  if  applica¬ 
tion  and  machine-dependent  functions  are  mixed  in  the  same  module  and  if 
processing  is  data  volume  limited.  As  an  example,  assume  the  measurements 
were  applied  during  the  design  phase  and  a  value  of  0.65  was  measured. 
Inserting  this  value  in  the  normalization  function  results  in  a  predicted 
rating  for  flexibility  of  .33  as  identified  by  point  A  in  Figure  19.  If 
the  Development  Manager  had  specified  a  rating  of  0.2  which  is  identified 
by  point  B,  he  has  an  indication  that  the  software  development  is  pro¬ 
gressing  well  with  respect  to  this  desired  quality. 

Calculate  Confidence  in  Quality  Assessment.  Using  statistical  tech¬ 
niques  a  level  of  confidence  can  be  calculated.  The  calculation  is  based 
on  the  standard  error  of  estimate  for  the  normalization  function  and  can 
be  derived  from  a  normal  curve  table  found  in  most  statistics  texts.  An 
example  of  the  deviation  process  is  shown  in  Figure  20  for  the  situation 
described  above.  Here  it  is  shown  that  the  Development  Manager  has  an 
86  percent  level  of  confidence  that  the  flexibility  of  the  system  will 
be  better  than  the  specified  rating. 


Reporting  Assessment  Results 

Each  of  the  preceding  steps  described  in  this  section  are  easily  auto¬ 
mated.  If  the  metrics  are  applied  automatically  then  the  metric  data  is 
available  in  machine  readable  form.  If  the  worksheets  are  applied 
manually,  then  the  data  can  be  entered  into  a  file,  used  to  calculate  the 
metric,  and  formatted  into  the  measurement  matrix  format.  The  automation 
of  the  analyses  involves  simple  matrix  manipulations.  The  results  of  the 
analyses  should  be  reported  at  various  levels  of  detail.  The  formats  of 
the  reports  are  left  to  the  discretion  of  the  quality  assurance  organiza¬ 
tion.  The  content  of  the  reports  to  the  different  managers  is  recommended 
in  the  following  paragraphs. 
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Table  XXI.  Normal itati on  Unctions 


reliabiut'  (design) 

MULTIVARIATE  .18M_t  .  ♦  .19  M.. 

function  1  41  • 3 

(  ET. 1  Error  Tolerance  Checklist 
ISI.3  Complexity  Measure 

INDIVIDUAL  .34  . 

FUNCTIONS  ,, 

•”  Msr . 3 

RELIABILITY  (IMPLEMENTATION) 

MULTIVARIATE  .48M_  ,*  .14M„.  , 

FUNCTION  tT*  il‘ 

ET. I  Error  Tolerance  Checklist 

[MOtVtOUAL  .57  «_  , 

•“  "SI.l 

U  HSt.J 
“  »SJ.4 

51. 3  Complexity  Measure 

SI.l  Design  Stricture  Measure 

51. 4  Coding  Simolicity  Measure 

MAINTAINABILITY  (OESIGN) 

INCIVIOUAL  .57  M-.  , 

FUNCTION  J513 

•”  MSU 

SI. 3  Complexity  Measure 

SI.l  Oesign  Structure  Measure 

MAINTAINABILITY  ( IMPLEMENTATION ) 

MULTIVARIATE  -,2*.61  M,..  ,♦  .l4MM0  ,*.33en  , 
FUNCTION  5‘-3  ,u,z  S0'2 

SI. 3  Complexity  Measure 

INDIVIDUAL 

FUNCTIONS  2.1  M$I  3 

*71  *S0.2 
•«  *S0.3 
•s  MSI.I 
•4  *SI.4 

•  V*  t  <1LUU  1  Qf  llUfJ  1  CIIICIU.3  L  1  UH 

Measure 

SO. 2  Effectiveness  of  Comments 
Measure 

SO  3  Oescriptiveness  of 
Implementation 

Language  Measure 

SI.l  Oesign  Structure 

Measure 

SI. 4  Coding  Simolicity  Measure 

FLEXIBILITY  (DESIGN) 

INOIVtOUAL 

FUNCTIONS  *ST  ,MMO  2 

36  V? 

MO. 2  Modular  Implementatir n 

GE.2  Generality  Checklist 

(Continued) 


FLEXIBILITY  ( IMPLEMENTATION) 

MULTIVARIATE  .22%.  .44Mr.  ,-.Q9M.n  . 

FUNCTION  Gt-2  S0-3 

-• 

INOIVIOUAL 

FUNCTIONS  .SM^  £ 

,72MG£.2 

,S9MS0.2 

•56"s0.3 

MO. 2  Modular  Implementation 
Measure 

GE.2  Generality  Checklist 

50. 2  Effectiveness  of  Comments 
Measure 

50. 3  Descriptiveness  of 
Implementation 

Language  Measure 

PORTABILITY  ( IMPLEMENTATION ) 

MULTIVARIATE  -J.7*.19M„  r  .76M_n  ,*2.5M„ 

rUNCTICN  50.1  50.2  50 

.Jf-54M:il.l 

INOIVIOUAL 

FUNCTIONS  !.07Msl  j 

1JMMI.I 

,,5MS0.Z 

50. 1  Quantity  of  Comments 

50. 2  Effectiveness  of 

Comments  Measure 

50. 3  Oescriptiveness  of 
Implementation 

Language  Measure 

MI.l  Machine  tndeoendenca 
Measure 

SI.l  Design  Structure 

Measure 
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Figure  19*  Norwallzatlon  Function  for  Flexibility  During  Design 


Report  to  the  Acquisition  Manager/Development  Manager.  The  report 
content  to  the  Development  Manager  should  provide  summary  information 
about  the  progress  of  the  development  toward  the  quality  goals  identified 
at  the  beginning  of  the  project. 

For  example,  if  ratings  were  specified  for  several  quality  factors, 
the  current  predicted  ratings  should  be  reported. 

PREDICTED  RATING 

QUALITY  GOALS  BASED  ON  DESIGN  DOCUMENT 

RELIABILITY  .9  .8 

MAINTAINABILITY  .8  .95 

If  specific  ratings  were  not  identified  but  the  important  qualities 
were  identified,  a  report  might  describe  the  percentage  of  modules  that 
currently  are  judged  to  be  below  the  average  quality  (as  a  result  of  the 
sensitivity  analysis)  or  that  are  below  a  specified  threshold  value  (as 
a  result  of  the  threshold  analysis).  These  statistics  provide  a  progress 
status  to  the  manager.  Further  progress  status  is  indicated  by  reporting 
the  quality  growth  of  the  system  or  of  individual  modules.  The  quality 
growth  is  depicted  by  reporting  the  scores  achieved  during  the  various 
phases  of  development.  Ultimately  the  ratings  should  progressively 
score  higher  than  those  received  during  requirements.  This  progress  is 
based  on  the  identification  of  problems  in  the  early  phases  which  can 
then  be  corrected. 

Reports  to  Quality  Assurance  Manager.  In  addition  to  the  summary 
quaiity  progress  reports,  the  quality  assurance  manager  and  his  staff  will 
want  detailed  metric  reports.  These  reports  will  provide  all  of  the 
results  of  the  Analyses  and  perhaps  provide  the  measurement  matrix  itself 
for  examinations.  In  addition  to  the  detailed  reports,  the  quality 
assurance  manager  should  be  provided  with  reports  on  the  status  of  the 
application  of  the  metrics  themselves  by  the  quality  assurance  staff. 

These  status  reports  will  provide  information  on  total  number  of  modules 
and  the  number  which  inspectors  have  analyzed. 

Reports  to  the  Development  Team.  The  development  team  should  be 
provided  detailed  information  on  an  exception  basis.  This  information  is 
derived  from  the  analyses.  Examples  of  the  information  would  be  quality 
problems  that  have  been  identified,  which  characteristics  or  measurements 
of  the  software  products  are  poor,  and  which  modules  have  been  identified 
as  requiring  rework.  These  exception  reports  should  contain  the  details 
of  why  the  assessment  revealed  them  as  potential  problems.  It  is  based 
on  this  information  that  corrective  actions  will  be  taken. 
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APPENDIX  K 

SAMPLE  OF  AMT  REPORTS 


(Ref  88) 


work  _>mi  :  r  hcpoh  r 


fie  Auiksfi^et  report  displays  the  r.  iw  data  entered  in  each  worksheet.  It 
rtprisentb  the  current  values  in  the  data  base.  It  is  used  to  verify  and 
track,  data  entry. 


AUTOMATED  MEASUREMENT  TOOL 
WORKSHEET  REPORT 
WORKSHEET  3 


DATA  3ase  Amtexs 

MOOUIE:  EXSGET  OATE:  12/23/81 


I.  STRUCTURE  (RELIABILITY,  MAINTAINABILITY,  TESTABILITY) 

1.  NUMBER  OF  LINES  OF  CODE  95. 

2.  NUMBER  OF  LINES  EXCLUDING  COMMENTS  47. 

3.  NUMBER  OF  MACHINE  LEVEL  LANGUAGE  STATEMENTS  0. 

4.  NUMBER  OF  DECLARATIVE  STATEMENTS  4. 

5.  NUMBER  OF  DATA  MANIPULATION  STATEMENTS  5. 

6.  NUMBER  OF  STATEMENT  LABELS  (EXCLUDING  FORMAT  STATEMENTS  0. 

^7.  NUMBER  OF  ENTRANCES  INTO  MOOUIE  1. 

ENTER  [CR]  TO  CONTINUE  'E*-TO  EXIT: 

8.  NUMBER  OF  EXISTS  FROM  MOOULE  2. 

9.  MAXIMUM  NESTING  LEVEL  3. 

10.  NUMBER  OF  DECISION  POINTS  (IF,  WHILE,  REPEAT,  DO,  CASE)  10. 

11.  NUMBER  OF  SUB-DECISION  POINTS  0. 

12.  NUMBER  OF  CONDITIONAL  BRANCHES  (COMPUTED  TO  GO  6. 

13.  NUMBER  OF  UNCONDITIONAL  8RANCHES  (GOTO,  ESCAPE)  0. 

14.  NUMBER  OF  LOOPS  (WHILE,  DO)  4. 

15.  NUMBER  OF  LOOPS  WITH  JUMPS  OUT  OF  LOOPS  0. 

16.  NUMBER  OF  LOOPS  INDICIES  THAT  ARE  MODIFIED  0. 

17.  NUMBER  OF  MOOULE  MODIFICATIONS  (SWITH,  ALTER)  0. 

18.  NUMBER  OF  NEGATIVE  OR  COMPLICATED  COMPOUND  BOOLEAN  EXPRESSIONS  0. 

19.  IS  A  STRUCTURED  LANGUAGE  USED?  YES 

20.  IS  FLOW  TOP  TO  BOTTOM  (ABSENSE  OF  BACKWARD  BRANCHING  GOTO's)?  YES 


II.  CONCISENESS  (MAINTAINABILITY) 

1.  NUMBER  OF  OPERATORS 

2.  NUMBER  OF  UNIQUE  OPERATORS 

3.  NUMBER  OF  OPERANOS 

4.  NUMBER  OF  UNIQUE  OPERANDS 
ENTER  (CR)  TO  CONTINUE,  'E'  TO 


4. 

1. 

8. 

3. 

EXIT: 


7  ip. 
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EXCEPTION  REPORT 


The  exception  report  delivers  the  relationship  of  each  module  to  a  given 
threshold  value  of  a  particular  metric.  The  relationship  (less  than,  equal 
to,  or  greate-  then)  and  the  threshold  value  is  input  from  the  user.  This 
report  can  be  used  to  Identify  modules  whose  scores  do  not  meet  a  certain 
threshold,  identifying  them  as  potential  problems. 


AUTOMATED  MEASUREMENT  TOOL 
EXCEPTIONS  REPORT 


0ATA6ASE:  AMTEXS  DATE:  12/23/81 

METRIC:  ET.  2 

PHASE:  MOOULE  IMPLEMENTATION 
THRESHOLD  VALUE:  0.65 
RELATION:  LESS  THAN 

THE  FOLLOWING  MOOULES  ARE  WITHIN  RANGE  REQUESTED 


MODULE  NAME 

VALUE 

EXSCEX 

0. 

EXCDLP 

0.500 

EXSDBG 

0.333 

EXSHLP 

0. 

EXSPGfi 

0. 

EXSUPK 

0. 
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NORMALIZATION  REPORT 


The  Norma  1 i z  a  t ion  Report  provides  the  user  with  the  overall  rating  of  a 
selected  quality  factor.  A  series  of  regression  eouations  are  displayed  whi;h 
have  been  empirically  derived  from  research.  The  current  metric  values  a-e 
substituted  in  the  equations  and  a  rating  for  the  selected  quality  factor  is 
calculated.  Regression,  equations  exist  for  the  quality  factors  reliabl  lity, 
maintainability,  portability,  and  flexibility  only: 

AUTOMATED  MEASUREMENT  TOOL 
NORMALIZATION  FUNCTION  REPORT 


DATABASE:  AMTEXS 

MOOULE:  EXSGET  OATE:  12/23/81 


OESIGN  NORMALIZATION  FUNCTION  IMPLEMENTATION  NORMALIZATION  FUNCTION 

FACTOR:  PORTABILITY 

#  NO  DESIGN  NORMALIZATION  FUNCTION  PORTABILITY  »  -1.7  +  .19  (SD.l)  ♦ 

FOR  PORTABILITY  FACTOR  .76(SD.2)  +  2.5(SD.3)  ♦  .64(MI.l) 

SD.l  *  0.426 
SD.2  »  0.857 
SD.3  *  1.000 
MI . 1  »  0.972 


PORTABILITY  =  2,154 


METRIC  REPORT 


This  report  calculates  the  value  of  each  metric  catagorized  by  factor  and  by 
development  phase.  This  report  is  used  to  d<  t^rmine  a  total  picture  of  the 
project  as  measurements  are  taken. 

AUTOMATED  MEASUREMENT  TOOL 
METRIC  REPORT/MODULE  IMPLEMENTATION  PHASE 


DATABASE:  AMTEXS 


MOOULE:  EXSGET 

DATE: 

12/23/81 

FACTOR 

CRITERIA 

METRIC 

VALUE 

CORRECTNESS 

Traceability 

TR.1 

1.000 

Completeness 

CP.l 

0.667 

Consistency/Procedure 

CS.l 

1.000 

Consistency/Data 

CS.2 

0.500 

RELIABILITY 

Consistency/Procedure 

CS.l 

1 .000 

Consistency/Data 

CS.2 

0.500 

Accuracy 

AY.  1 

1.000 

Error  Tolerance/Control 

ET.l 

1.000 

Error  Tolerance/Input  Data 

ET.2 

1.000 

Error  Tol ./Computational  Fail. 

ET.3 

0. 

Desian  Structure 

SI- 1 

0.625 

Complexity 

SI. 3 

0.100 

Code  Simplicity 

SI. 4 

0.722 

MAINTAINABILITY 

Cons istency/procedure 

CS.l 

1.000 

Consistency/Oata 

CS.2 

0.500 

Design  Structure 

SI.l 

0.625 

Complexity 

SI. 3 

0.100 

Code  Simplicity 

SI. 4 

0.722 

Modular  Implementation 

MO. 2 

0.750 

Quantity  of  Comments 

SO.  1 

0.426 

Effectiveness  of  Comments 

SD.2 

0.857 

Conciseness 

C0.1 

1.000 

TESTABILITY 

Oesign  Structure 

SI.l 

0.625 

Complexity 

SI. 3 

0. 100 

Code  Simpl Icity 

SI. 4 

0.722 

Modular  Implementation 

MO. 2 

0.750 

Quantity  of  Comments 

S0.1 

0.426 

Effectiveness  of  Comments 

SD.2 

0.857 

Descriptiveness  of  Impl.  Lang. 

SO. 3 

1 .000 

portability 

Modular  Implementation 

MO. 2 

0.750 

Quantity  of  Comments 

SD.  1 

0.486 

I 


FACTOR 


REUSABILITY 


FLEXIBILITY 


INTEROPERABILITY 

EFFICIENCY 


CRITERIA 

METRIC 

VALIJi: 

Effectiveness  of  Comments 

SD.2 

O.P57 

Descript iveness  of  Impl.  Lang. 

SD.3 

1 .000 

System  Software/ Independence 

SS.l 

0.500 

Machine  Independence 

MI .  1 

0.972 

Modular  Implementation 

MO. 2 

0.750 

General ity/ Implementation 

GE.2 

0.750 

Quantity  of  Comments 

SO .  1 

0.426 

Effectiveness  of  Comments 

SD.2 

0.857 

Descriptiveness  of  Impl.  Lang. 

SD.3 

1.000 

System  Software/ Independence 

SS.l 

0.500 

Machine  Independence 

MI .  1 

0.972 

Modular  Implementation 

MO. 2 

0.750 

Generality/ Implementation 

GE.2 

0.750 

Oata  Storage  Expansion 

EX.  1 

0. 

Computational  Extensibility 

EX. 2 

0.500 

Quantity  of  Comments 

S0.1 

0.426 

Effectiveness  of  Comments 

SD.2 

0.857 

De script Iveness  of  Impl  Lang. 

SO. 3 

1.000 

Modular  Implementation 

MO. 2 

0.750 

Iterative  Processing 

EE. 2 

1.000 

Oata  Usage 

EE. 3 

0.668 

STATISTICS  REPORT 


The  Statistics  Report  provides  a  profile  of  COBOL  onstmcts  for  each  modul 

AUTOMATED  MEASUREMENT  TOOL 
STATISTICS  REPORT 

DATABASE:  AMTEXS 

MOOULE:  EXSGET  DATE:  12/23/81 

NUMBER  OF  LINES  OF  CODE  95. 

NUMBER  OF  PERFORM  STATEMENTS  4. 

NUMBER  OF  EXTERNAL  CALLS  0. 

NUMBER  OF  EXECUTABLE  STATEMENTS  (PROCECURE  DIVISION)  43. 

NUMBER  OF  COMMENTS  48. 

NUMBER  OF  DECLARATIONS  (DATA  DIVISION)  4. 

NUMBER  OF  LABELS  0. 

NUMBER  OF  I/O  REFERENCES  6. 

NUM8ER  OF  REDEFINES  (EQUIVALENTS)  0. 

NUMBER  OF  LEVEL  88  DATA  ITEMS  (LOCAL  VARIABLES)  1. 


SUMMARY  REPORT 


The  summary  report  provides  a  summary  of  the  metric  scores  for  a!!  of  the 
modules  in  the  system. 

AUTOMATED  MEASUREMENT  TOOL 
METRIC  SUMMARY  REPORT 

DATABASE:  AMTEXS  DATE:  12/23/81 

MODULE:  EXSGET 

AY. 1  -  1.000 
CS.2  «  0.500 
ET.2  *  1.000 
GE.2  •  0.750 
SO. 2  -  0.857 
SI. 4  «  0.722 


Y'A 


CO.  I  «  1.000 
EE. 2  -  1.000 
ET.3  =>  0. 
MI.l  -  0.972 
SO. 3  =*  1.000 
SS.l  =  0.500 


CP.l  «  0.667 
EE. 3  -  0.668 
EX. 1  -  0. 

MO. 2  *  0.750 
SI.l  •  0.625 
TR.l  «  1.000 


CS.1  *  1.000 
ET.l  *  1.000 
EX. 2  «  0.500 
SD.l  =  0.426 
SI. 3  *  0.100 


QUALITY  GROWTH  REPORT 


When  the  user  wishes  to  track  the  value  of  a  particular  metric  over  time,  the 
Quality  Growth  Report  will  furnish  a  tabular  display  of  the  scores  of  a 
selected  metric  over  the  plhases  of  the  project.  This  report  is  used  to  track 
a  particular  metric  through  a  project  to  see  how  its  value  changes. 

AUTOMATED  MEASUREMENT  TOOL 
QUALITY  GROWTH  REPORT 


OAT ABASE:  AMTEXS 

MOOULE:  EXSGET  DATE:  12/23/81 


METRIC 


DETAILED  MODULE 

DESIGN  IMPLEMENTATION 


ET.2 


0.750 


1.000 


5-s 


MATRIX  REPORT 


This  report  displays  the  average  and  standard  deviations  for  all  metric  /alues 
modules.  This  report  displays  all  of  this  information  in  a  matrix  form 
allowing  the  user  to  easily  identify  modules  with  metric  scores  that  vary  from 
the  system  average. 


AUTOMATED  MEASUREMENT  TOOL 
MATRIX  REPORT 


OATABASE:  AMTEXS 
PAGE  *  1 


MODULE  NAME 

AY.  1 

C0.1 

EXSCEX 

0. 

1.000 

EXSCHK 

1 .000 

1.000 

EXSCLP 

1.000 

1.000 

EXSOBG 

0. 

1.000 

EXSGET 

1.000 

1.000 

EXSHLP 

0. 

1.000 

EXSPGR 

0. 

1.000 

EXSQRY 

0. 

1.000 

EXSSSM 

0. 

1.000 

EXSUPK 

0. 

1.000 

AVERAGE  « 

0.300 

0.900 

STANDARD  DEVIATION  * 

0.438 

0.316 

DATE : 

12/23/81 

CP.l 

CS.l 

CS.2 

EE. 2 

0. 

0. 

0. 

1.000 

0.667 

1.000 

0.500 

1.000 

0.667 

1.000 

0.500 

1.000 

0. 

0. 

0. 

0. 

0.667 

1.000 

0.500 

1  000 

0.833 

1.000 

0.500 

0. 

1.000 

1.000 

0.500 

1.000 

0.667 

1.000 

0.500 

0. 

1.000 

1.000 

0.500 

0. 

0.625 

1.000 

0.500 

1 .000 

0.550 

0.700 

0.350 

0.500 

0.401 

0.483 

0.242 

0.527 

i 


MODULE  PE PORT 


This  report  displays  the  catalog  of  modules  that  -have  been  entered  into  t he 
database.  It  providss  a  status  report  on  the  database. 
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OATABASE:  AMTEXS 


12/23/81 


WS1  CONTAINS  SOME  NIL  VALUES 
WS2A  CONTAINS  SOME  NIL  VALUES 

THE  FOLLOWING  MOOULES  ARE  PRESENTLY  IN  THE  CURRENT  DATABASE: 


1.  EXSCEX  ** 
3.  EXSCLP  * 
5.  EXSGET  * 
7.  EXSPGR  * 
9.  EXSSSM  * 


2.  EXSCHK  * 
4.  EXSDBG  ** 
6.  EXSHLP  * 
8.  EXSQRY  * 
10.  EXSUPK  * 


NOTE: 


TOTAL  NUMBER  OF  MOOULES  IN  DATABASE  IS  10. 

INDICATES  BOTH  WS28  ANO  WS3  CONTAIN  SOME  NIL  VALUES. 
IND  'ATES  WS2B  CONTAINS  SOME  NIL  VALUES. 
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